Deep Learning in Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning in Python\n",
    "- neural networks\n",
    "- deep learning models\n",
    "- using Keras 2.0\n",
    "- with Dan Becker from Kaggle, Keras and TensorFlow libraries\n",
    "\n",
    "Applications: robotics, NLP, image recognition, AI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Basics of Deep Learning and Neural Networks\n",
    "Interactions\n",
    "- Neural Networks account for interactions really well\n",
    "- Deep learning uses powerful neural networks\n",
    "    - text, images, videos, audio, source code\n",
    "- Deep learning models capture interactions\n",
    "    - can take 2 features and caculate this interaction to predict the outcome\n",
    "    - reality is that there are a lot of interactions\n",
    "- Structure\n",
    "    - Input Layer - features\n",
    "    - Output Layer - predicted\n",
    "    - Hidden Layer - everything in between input and output\n",
    "        - each component is a NODE\n",
    "        - we don't have data and can't observe this layer directly\n",
    "        - the more nodes, the more interactions we capture\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Forward propagation = using data to make predictions\n",
    "Example: bank transactions\n",
    "- make predictions based on:\n",
    "    - number of children\n",
    "    - number of existing accounts\n",
    "- customer with 2 children, 3 accounts\n",
    "    - line from input to each node\n",
    "    - each line has a weight for how strongly the input affects the hidden node \n",
    "    - first set of weights - parameters we train/change when NN fit to data\n",
    "- Dot product\n",
    "    - hidden node prediction = sum(input*weight)\n",
    "    - output node - repeat the same multiply-add process (sum(node*weight))\n",
    "\n",
    "Forward propagation = input to hidden node to output\n",
    "- dot product (multiply-add process)\n",
    "- forward propagation for one data point at a time\n",
    "- output is the prediction for that data point\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forward propagation code\n",
    "\n",
    "import numpy as np\n",
    "# recall 2 children, 3 existing accounts\n",
    "input_data = np.array([2,3])\n",
    "# 2 hidden nodes\n",
    "weights = {'node_0': np.array([1,1]),\n",
    "           'node_1': np.array([-1,1]),\n",
    "           'output': np.array([2,-1])}\n",
    "node_0_value = (input_data * weights['node_0']).sum()\n",
    "node_1_value = (input_data * weights['node_1']).sum()\n",
    "\n",
    "hidden_layer_values = np.array([node_0_value, node_1_value])\n",
    "print(hidden_layer_values)\n",
    "# [5,1]\n",
    "\n",
    "# get output\n",
    "output = hidden_layer_values * weights['output']).sum()\n",
    "print(output)\n",
    "# 9\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Coding the forward propagation algorithm\n",
    "In this exercise, you'll write code to do forward propagation (prediction) for your first neural network:\n",
    "\n",
    "<img src=\"image1.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/1_4.png)\n",
    "\n",
    "Each data point is a customer. The first input is how many accounts they have, and the second input is how many children they have. The model will predict how many transactions the user makes in the next year. You will use this data throughout the first 2 chapters of this course.\n",
    "\n",
    "The input data has been pre-loaded as input_data, and the weights are available in a dictionary called weights. The array of weights for the first node in the hidden layer are in weights['node_0'], and the array of weights for the second node in the hidden layer are in weights['node_1'].\n",
    "\n",
    "The weights feeding into the output node are available in weights['output'].\n",
    "\n",
    "NumPy will be pre-imported for you as np in all exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate node 0 value: node_0_value\n",
    "node_0_value = (input_data * weights['node_0']).sum()\n",
    "\n",
    "# Calculate node 1 value: node_1_value\n",
    "node_1_value = (input_data * weights['node_1']).sum()\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_value, node_1_value])\n",
    "\n",
    "# Calculate output: output\n",
    "output = (hidden_layer_outputs * weights['output']).sum()\n",
    "\n",
    "# Print output\n",
    "print(output)\n",
    "\n",
    "# -39"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Activation functions\n",
    "- to achieve max predictive power, we need activation function in hidden layer\n",
    "- allows model to capture non-linearities\n",
    "- applied to node inputs to produce node output\n",
    "\n",
    "Improving our neural network\n",
    "- used to use tanh\n",
    "- now ReLU is the standard\n",
    "\n",
    "ReLU (Rectified Linear Activation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# activation functions with tanh\n",
    "\n",
    "import numpy as np\n",
    "input_data = np.array([-1, 2])\n",
    "weights = {'node_0': np.array([3,3]),\n",
    "           'node_1': np.array([1,5]),\n",
    "           'output': np.array([2,-1])}\n",
    "\n",
    "# note distinguishing input and output\n",
    "node_0_input = (input_data * weights['node_0']).sum()\n",
    "node_0_output = np.tanh(node_0_input)\n",
    "node_1_input = (input_data * weights['node_1']).sum()\n",
    "node_1_output = np.tanh(node_1_input)\n",
    "\n",
    "hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "output = hidden_layer_outputs * weights['output']).sum()\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.a ReLU - The Rectified Linear Activation Function\n",
    "As Dan explained to you in the video, an \"activation function\" is a function applied at each node. It converts the node's input into some output.\n",
    "\n",
    "The rectified linear activation function (called ReLU) has been shown to lead to very high-performance networks. This function takes a single number as an input, returning 0 if the input is negative, and the input if the input is positive.\n",
    "\n",
    "Here are some examples:\n",
    "- relu(3) = 3 \n",
    "- relu(-3) = 0 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ReLU\n",
    "\n",
    "def relu(input):\n",
    "    '''Define your relu activation function here'''\n",
    "    # Calculate the value for the output of the relu function: output\n",
    "    # return 0 if input is negative, else return input\n",
    "    output = max(input, 0)\n",
    "    \n",
    "    # Return the value just calculated\n",
    "    return(output)\n",
    "\n",
    "# Calculate node 0 value: node_0_output\n",
    "node_0_input = (input_data * weights['node_0']).sum()\n",
    "node_0_output = relu(node_0_input)\n",
    "\n",
    "# Calculate node 1 value: node_1_output\n",
    "node_1_input = (input_data * weights['node_1']).sum()\n",
    "node_1_output = relu(node_1_input)\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "\n",
    "# Calculate model output (do not apply relu)\n",
    "model_output = (hidden_layer_outputs * weights['output']).sum()\n",
    "\n",
    "# Print model output\n",
    "print(model_output)\n",
    "\n",
    "# 52\n",
    "# we predicted 52 transactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.b Applying the network to many observations/rows of data\n",
    "You'll now define a function called predict_with_network() which will generate predictions for multiple data observations, which are pre-loaded as input_data. As before, weights are also pre-loaded. In addition, the relu() function you defined in the previous exercise has been pre-loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define predict_with_network()\n",
    "def predict_with_network(input_data_row, weights):\n",
    "    \"\"\"accepts two arguments - input_data_row and weights - and \n",
    "    returns a prediction from the network as the output.\"\"\"\n",
    "        # Calculate node 0 value\n",
    "    node_0_input = (input_data_row * weights['node_0']).sum()\n",
    "    node_0_output = relu(node_0_input)\n",
    "\n",
    "    # Calculate node 1 value\n",
    "    node_1_input = (input_data_row * weights['node_1']).sum()\n",
    "    node_1_output = relu(node_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_layer_outputs\n",
    "    hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "    \n",
    "    # Calculate model output\n",
    "    input_to_final_layer = (hidden_layer_outputs * weights['output']).sum()\n",
    "    model_output = relu(input_to_final_layer)\n",
    "    \n",
    "    # Return model output\n",
    "    return(model_output)\n",
    "\n",
    "\n",
    "# Create empty list to store prediction results\n",
    "results = []\n",
    "for input_data_row in input_data:\n",
    "    # Append prediction to results\n",
    "    results.append(predict_with_network(input_data_row, weights))\n",
    "\n",
    "# Print results\n",
    "print(results)\n",
    "\n",
    "# [52, 63, 0, 148]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Deeper networks\n",
    "Multiple hidden layers\n",
    "- can scale to even 1000 layers\n",
    "- same propogation process\n",
    "- assume all layers use ReLU activation function\n",
    "\n",
    "Representation learning (aka deep learning)\n",
    "- deep networks internally build representations of patterns in the data\n",
    "- partially replace the need for feature engineering\n",
    "- called representation learning b/c subsequent layers build increasingly sophisticated representations of raw data until we get to a stage to make predictions\n",
    "- example in images:\n",
    "    - early nodes: diagonal line\n",
    "    - later node: face node\n",
    "    - next node: cat node    \n",
    "- So, more complex or \"higher level\" interactions are in the last layers of the model\n",
    "\n",
    "Deep learning\n",
    "- pro: modeler doesn't need to specify the interactions\n",
    "- when you train the model, the neural network gets weights that find the relevant patterns to make better predictions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.a Multi-layer neural networks\n",
    "In this exercise, you'll write code to do forward propagation for a neural network with 2 hidden layers. Each hidden layer has two nodes. The input data has been preloaded as input_data. The nodes in the first hidden layer are called node_0_0 and node_0_1. Their weights are pre-loaded as weights['node_0_0'] and weights['node_0_1'] respectively.\n",
    "\n",
    "The nodes in the second hidden layer are called node_1_0 and node_1_1. Their weights are pre-loaded as weights['node_1_0'] and weights['node_1_1'] respectively.\n",
    "\n",
    "We then create a model output from the hidden nodes using weights pre-loaded as weights['output'].\n",
    "\n",
    "<img src=\"image2.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch1ex10.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# forward propagation with 2 hidden layers\n",
    "\n",
    "def predict_with_network(input_data):\n",
    "    # Calculate node 0 in the first hidden layer\n",
    "    node_0_0_input = (input_data * weights['node_0_0']).sum()\n",
    "    node_0_0_output = relu(node_0_0_input)\n",
    "\n",
    "    # Calculate node 1 in the first hidden layer\n",
    "    node_0_1_input = (input_data * weights['node_0_1']).sum()\n",
    "    node_0_1_output = relu(node_0_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_0_outputs\n",
    "    hidden_0_outputs = np.array([node_0_0_output, node_0_1_output])\n",
    "    \n",
    "    # Calculate node 0 in the second hidden layer\n",
    "    node_1_0_input = (hidden_0_outputs * weights['node_1_0']).sum()\n",
    "    node_1_0_output = relu(node_1_0_input)\n",
    "\n",
    "    # Calculate node 1 in the second hidden layer\n",
    "    node_1_1_input = (hidden_0_outputs * weights['node_1_1']).sum()\n",
    "    node_1_1_output = relu(node_1_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_1_outputs\n",
    "    hidden_1_outputs = np.array([node_1_0_output, node_1_1_output])\n",
    "\n",
    "    # Calculate model output: model_output\n",
    "    model_output = (hidden_1_outputs * weights['output']).sum()\n",
    "    \n",
    "    # Return model_output\n",
    "    return(model_output)\n",
    "\n",
    "output = predict_with_network(input_data)\n",
    "print(output)\n",
    "\n",
    "# 182\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Optimize a Neural Network with backward propagation\n",
    "Consider a baseline neural network\n",
    "- uses an identity function (returns the input)\n",
    "- given input and true value of target\n",
    "- use forward propagation to fill in values for hidden layer node values\n",
    "- use hidden layer node values to make a prediction on output, then compute error compared to actual value of target\n",
    "- changing the weights of the hidden layers improves accuracy of the predicted target\n",
    "\n",
    "Predictions with multiple points\n",
    "- making accurate predictions gets harder with more points\n",
    "- at any set of weights, there are many values of the error\n",
    "    - ...corresponding to the many points we make predictions for\n",
    "    \n",
    "Loss function\n",
    "- aggregates erros in predictions from many data points into single number\n",
    "- measure of model's predictive performance\n",
    "- example: column loss for regression - mean squared error loss function\n",
    "- Lower loss function value means a better model\n",
    "- Goal: Find the weights that give the lowest value for the loss function\n",
    "- How? use Gradient descent algorithm\n",
    "\n",
    "Gradient descent\n",
    "- analogy: imagine you are in a pitch dark field\n",
    "    - want to find the lowest point\n",
    "    - feel the ground to see how it slopes\n",
    "    - take a small step downhill\n",
    "    - repeat until it is uphill in every direction\n",
    "- steps:\n",
    "    - start at random point\n",
    "    - until you are somewhere flat:\n",
    "        - find the slope (the derivative)\n",
    "        - take a step downhill'\n",
    "\n",
    "Optimizing a model with a single weight\n",
    "- if slope is positive, to find the low point go in the opposite direction towards lower numbers\n",
    "- eventually get to the minimum value\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Calculating model errors\n",
    "For the exercises in this chapter, you'll continue working with the network to predict transactions for a bank.\n",
    "\n",
    "What is the error (predicted - actual) for the following network when the input data is [3, 2] and the actual value of the target (what you are trying to predict) is 5? It may be helpful to get out a pen and piece of paper to calculate these values.\n",
    "\n",
    "![example](https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch2_ex2_3.png)\n",
    "\n",
    "Answer:\n",
    "output = 16\n",
    "actual value = 5\n",
    "error = 16 - 5 = 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Understanding how weights change model accuracy\n",
    "Imagine you have to make a prediction for a single data point. The actual value of the target is 7. The weight going from node_0 to the output is 2, as shown below. If you increased it slightly, changing it to 2.01, would the predictions become more accurate, less accurate, or stay the same?\n",
    "<img src=\"image3.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch2_ex2_3.png\n",
    "\n",
    "Answer: less accurate\n",
    "Increasing weight from 2 to 2.01 would increase resulting error from 9 to 9.08. So less accurate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Coding how weight changes affect accuracy\n",
    "Now you'll get to change weights in a real network and see how they affect model accuracy!\n",
    "\n",
    "Have a look at the following neural network: \n",
    "<img src=\"image4.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch2ex4.png\n",
    "\n",
    "Its weights have been pre-loaded as weights_0. Your task in this exercise is to update a single weight in weights_0 to create weights_1, which gives a perfect prediction (in which the predicted value is equal to target_actual: 3).\n",
    "\n",
    "Use a pen and paper if necessary to experiment with different combinations. You'll use the predict_with_network() function, which takes an array of data as the first argument, and weights as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The data point you will make a prediction for\n",
    "input_data = np.array([0, 3])\n",
    "\n",
    "# Sample weights\n",
    "weights_0 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 2],\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# The actual target value, used to calculate the error\n",
    "target_actual = 3\n",
    "\n",
    "# Make prediction using original weights\n",
    "model_output_0 = predict_with_network(input_data, weights_0)\n",
    "\n",
    "# Calculate error: error_0\n",
    "error_0 = model_output_0 - target_actual\n",
    "\n",
    "# Create weights that cause the network to make \n",
    "#  perfect prediction (3): weights_1\n",
    "weights_1 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 0], # changed 2nd weight to 0\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# Make prediction using new weights: model_output_1\n",
    "model_output_1 = predict_with_network(input_data, weights_1)\n",
    "\n",
    "# Calculate error: error_1\n",
    "error_1 = model_output_1 - target_actual\n",
    "\n",
    "# Print error_0 and error_1\n",
    "print(error_0)\n",
    "print(error_1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4: Scaling up to multiple data points\n",
    "You've seen how different weights will have different accuracies on a single prediction. But usually, you'll want to measure model accuracy on many points. You'll now write code to compare model accuracies for two different sets of weights, which have been stored as weights_0 and weights_1.\n",
    "\n",
    "input_data is a list of arrays. Each item in that list contains the data to make a single prediction. target_actuals is a list of numbers. Each item in that list is the actual value we are trying to predict.\n",
    "\n",
    "In this exercise, you'll use the mean_squared_error() function from sklearn.metrics. It takes the true values and the predicted values as arguments.\n",
    "\n",
    "You'll also use the preloaded predict_with_network() function, which takes an array of data as the first argument, and weights as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# Create model_output_0 \n",
    "model_output_0 = []\n",
    "# Create model_output_1\n",
    "model_output_1 = []\n",
    "\n",
    "# Loop over input_data\n",
    "for row in input_data:\n",
    "    # Append prediction to model_output_0\n",
    "    model_output_0.append(predict_with_network(row, weights_0))\n",
    "    \n",
    "    # Append prediction to model_output_1\n",
    "    model_output_1.append(predict_with_network(row, weights_1))\n",
    "\n",
    "# Calculate the mean squared error for model_output_0: mse_0\n",
    "mse_0 = mean_squared_error(target_actuals, model_output_0)\n",
    "\n",
    "# Calculate the mean squared error for model_output_1: mse_1\n",
    "mse_1 = mean_squared_error(target_actuals, model_output_1)\n",
    "\n",
    "# Print mse_0 and mse_1\n",
    "print(\"Mean squared error with weights_0: %f\" %mse_0)\n",
    "print(\"Mean squared error with weights_1: %f\" %mse_1)\n",
    "\n",
    "# Mean squared error with weights_0: 294.000000\n",
    "# Mean squared error with weights_1: 395.062500\n",
    "\n",
    "# model_output_1 has a higher mean squared error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Gradient descent\n",
    "- if slope is positive:\n",
    "    - going opposite the slope means moving to lower numbers\n",
    "    - subtract the slope from the current value\n",
    "    - problem: too big a step might lead us astray\n",
    "- Solution: learning rate\n",
    "    - update each weight by subtracting (learning rate * slope)\n",
    "    - learning rates frequently around 0.01\n",
    "    - this ensures small steps toward optimal rate\n",
    "\n",
    "Slope calculation example\n",
    "- to calculate the slope of a weight, need to multiply:\n",
    "    - slope of the loss function w.r.t. prediction (value at the node we feed into)\n",
    "        - = 2 * (Predicted value - Actual value) = 2* Error\n",
    "        - example: 2 * (6-10) = -8\n",
    "    - the value of the node that feeds into our weight\n",
    "        - input = 3\n",
    "    - slope of the activation function w.r.t. value we feed into\n",
    "        - this example does not have activation function, so skip\n",
    "    - slope of mean squared loss function = 2 * -4 * 3 = -24\n",
    "- if learning rate = 0.01, new weight updated to 2-0.01(-24) = 2.24\n",
    "- for multiple weights, repeat process for each weight separately, then update both weights simultaneously using their derivative\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Code to calculate slope and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "weights = np.array([1,2])\n",
    "input_data = np.array([3, 4])\n",
    "target = 6\n",
    "learning_rate = 0.01\n",
    "preds = (weights * input_data).sum()\n",
    "error = preds - target\n",
    "print(error)\n",
    "# 5\n",
    "\n",
    "# slope calculation\n",
    "gradient = 2 * input_data * error\n",
    "gradient\n",
    "# array([30, 40])\n",
    "\n",
    "# update weights by small step in that direction\n",
    "weights_updated = weights - learning_rate * gradient\n",
    "preds_updated = (weights_updated * input_data).sum()\n",
    "error_updated = preds_updated - target\n",
    "print(error_updated)\n",
    "# -2.5\n",
    "# improvement in error, repeating this would result in more improvements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Calculating slopes\n",
    "You're now going to practice calculating slopes. \n",
    "When plotting the mean-squared error loss function against predictions,\n",
    "- the slope is 2*x*(y-xb),\n",
    "- or 2*(input_data)*(error)\n",
    "\n",
    "Note that x and b may have multiple numbers (x is a vector for each data point, and b is a vector). In this case, the output will also be a vector, which is exactly what you want.\n",
    "\n",
    "You're ready to write the code to calculate this slope while using a single data point. You'll use pre-defined weights called weights as well as data for a single point called input_data. The actual value of the target you want to predict is stored in target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the predictions: preds\n",
    "preds = (input_data * weights).sum()\n",
    "\n",
    "# Calculate the error: error\n",
    "# This error corresponds to y-xb in the gradient expression\n",
    "error = target - preds\n",
    "\n",
    "# Calculate the slope: slope\n",
    "slope = 2 * input_data * error\n",
    "\n",
    "# Print the slope\n",
    "print(slope)\n",
    "\n",
    "# [-14 -28 -42]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Improving model weights (using the slope)\n",
    "Hurray! You've just calculated the slopes you need. Now it's time to use those slopes to improve your model. If you add the slopes to your weights, you will move in the right direction. However, it's possible to move too far in that direction. So you will want to take a small step in that direction first, using a lower learning rate, and verify that the model is improving.\n",
    "\n",
    "The weights have been pre-loaded as weights, the actual value of the target as target, and the input data as input_data. The predictions from the initial weights are stored as preds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate: learning_rate\n",
    "learning_rate = 0.01\n",
    "\n",
    "# Calculate the predictions: preds\n",
    "preds = (weights * input_data).sum()\n",
    "\n",
    "# Calculate the error: error\n",
    "error = preds - target\n",
    "\n",
    "# Calculate the slope: slope\n",
    "slope = 2 * input_data * error\n",
    "\n",
    "# Update the weights: weights_updated\n",
    "weights_updated = weights - (learning_rate * slope)\n",
    "\n",
    "# Get updated predictions: preds_updated\n",
    "preds_updated = (weights_updated * input_data).sum()\n",
    "\n",
    "# Calculate updated error: error_updated\n",
    "error_updated = preds_updated - target\n",
    "\n",
    "# Print the original error\n",
    "print(error)\n",
    "\n",
    "# Print the updated error\n",
    "print(error_updated)\n",
    "\n",
    "# 7\n",
    "# 5.04\n",
    "\n",
    "# Updating the model weights decreased the error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Making multiple updates to weights\n",
    "You're now going to make multiple updates so you can dramatically improve your model weights, and see how the predictions improve with each update.\n",
    "\n",
    "To keep your code clean, there is a pre-loaded get_slope() function that takes input_data, target, and weights as arguments. There is also a get_mse() function that takes the same arguments. The input_data, target, and weights have been pre-loaded.\n",
    "\n",
    "This network does not have any hidden layers, and it goes directly from the input (with 3 nodes) to an output node. Note that weights is a single array.\n",
    "\n",
    "We have also pre-loaded matplotlib.pyplot, and the error history will be plotted after you have done your gradient descent steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_updates = 20\n",
    "mse_hist = []\n",
    "\n",
    "# Iterate over the number of updates\n",
    "for i in range(n_updates):\n",
    "    # Calculate the slope: slope\n",
    "    # get_slope() is a pre-defined function\n",
    "    slope = get_slope(input_data, target, weights)\n",
    "    \n",
    "    # Update the weights: weights\n",
    "    # Learning rate = 0.01\n",
    "    weights = weights - 0.01 * slope\n",
    "    \n",
    "    # Calculate mse with new weights: mse\n",
    "    mse = get_mse(input_data, target, weights)\n",
    "    \n",
    "    # Append the mse to mse_hist\n",
    "    mse_hist.append(mse)\n",
    "\n",
    "# Plot the mse history\n",
    "plt.plot(mse_hist)\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Mean Squared Error')\n",
    "plt.show()\n",
    "\n",
    "# In the plot, you can see the mean squared error decreases as the \n",
    "# number of iterations go up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"image5.png\" width=\"500\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Backpropagation\n",
    "- use backpropagation to calculate the slopes you need to optimize more complex deep learning models\n",
    "- Backpropagation - takes error from output layer and propates through hidden layers to input\n",
    "- Allows gradient descent to update all weights in neural network (by getting gradients for all weights)\n",
    "- Important to understand the process, but you will generally use a library that implements this.\n",
    "\n",
    "Backpropoagation process\n",
    "- trying to estimate the slope of the loss function w.r.t. each weight\n",
    "- Do forward propagation to calculate predictions and errors, before you do backpropagation\n",
    "- Go back one layer at a time\n",
    "- Gradients for weight (to get slope) is product of:\n",
    "    1. Input node value - Node value feeding into that weight\n",
    "        - input node is given from data\n",
    "        - hidden nodes are calculated from forward propagation\n",
    "    2. Slope of loss function w.r.t. output node it feeds into\n",
    "    3. Slope of activation function at the node it feeds into\n",
    "- Need to also keep track of the slopes of the loss function w.r.t. node values\n",
    "- Slope of node values are the sum of the slopes for all weights that come out of them.\n",
    "\n",
    "ReLU activation function\n",
    "- slope = 0 if value is negatie\n",
    "- slope = 1 if value positive\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.a The relationship b/n forward and backward propagation\n",
    "If you have gone through 4 iterations of calculating slopes (using backward propagation) and then updated weights, how many times must you have done forward propagation?\n",
    "\n",
    "4\n",
    "\n",
    "Each time you generate predictions using forward propagation, you update the weights using backward propagation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.b Thinking about backward propagation\n",
    "If your predictions were all exactly right, and your errors were all exactly 0, the slope of the loss function with respect to your predictions would also be 0. In that circumstance, which of the following statements would be correct?\n",
    "\n",
    "The updates to all weights in the network would also be 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Backpropagation in practice\n",
    "Example:\n",
    "- node 0 value = 1\n",
    "- node 1 value = 3\n",
    "- weight 0 = 1\n",
    "- weight 1 = 2\n",
    "- output = 7, actual target value = 4\n",
    "    - error = 3\n",
    "    - relevant slope of output node = 2x error, 2*3 = 6\n",
    "- Top weight slope = 1*6 = 6\n",
    "- Bottom weight slope = 3*6 = 18\n",
    "- Calculate one layer back\n",
    "    - top node = 6\n",
    "    - bottom mode = 18\n",
    "    - \n",
    "Recall: Calculating slopes associated with any weight\n",
    "- Recall Gradients for weight is product of:\n",
    "    1. Input node value - Node value feeding into that weight\n",
    "        - input node is given from data\n",
    "        - hidden nodes are calculated from forward propagation\n",
    "    2. Slope of loss function w.r.t. output node it feeds into\n",
    "    3. Slope of activation function at the node it feeds into\n",
    "\n",
    "Backpropagation \n",
    "- Current weight value, Gradient\n",
    "    - 0, 0\n",
    "    - 1, 6\n",
    "    - 2, 0\n",
    "    - 3, 18\n",
    "    \n",
    "Backpropagation recap:\n",
    "- start at some random set of weights\n",
    "- use forward propagation to calculate the slope of the loss function w.r.t. each weight\n",
    "- multiply that slope by the learning rate, and subtract from the current weights\n",
    "- Repeat cycle until we get to a flat part\n",
    "\n",
    "Stochastic gradient descent\n",
    "- it's common to calculate slopes on only a subset of the data ('batch')\n",
    "- use a different batch of data to calculate the next update\n",
    "- once you've used all data, start over from the beginning\n",
    "- each time through the training data is called an epoch\n",
    "- when slopes are calculated on one batch at a time = stochastic gradient descent\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A round of backpropagation\n",
    "In the network shown below, we have done forward propagation, and node values calculated as part of forward propagation are shown in white. The weights are shown in black. Layers after the question mark show the slopes calculated as part of back-prop, rather than the forward-prop values. Those slope values are shown in purple.\n",
    "\n",
    "This network again uses the ReLU activation function, so the slope of the activation function is 1 for any node receiving a positive value as input. Assume the node being examined had a positive value (so the activation function's slope is 1).\n",
    "\n",
    "<img src=\"image6.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch2ex14_1.png\n",
    "\n",
    "Recall: Gradients for weight (to get slope) is product of:\n",
    "- Input node value\n",
    "    - 2\n",
    "- Slope of loss function w.r.t. output node it feeds into\n",
    "    - 3\n",
    "- Slope of activation function at the node it feeds into\n",
    "    - ReLU slope = 1\n",
    "\n",
    "What is the slope needed to update the weight with the question mark?\n",
    "\n",
    "<img src=\"image7.png\" width=\"500\" />\n",
    "Reference: https://s3.amazonaws.com/assets.datacamp.com/production/course_3524/datasets/ch2ex14_2.png\n",
    "\n",
    "Answer: 2x3 = 6, which is the weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Building Deep Learning models with keras\n",
    "Model building steps\n",
    "1. specify architecture\n",
    "    - how many layers\n",
    "    - how many nodes in each layer\n",
    "    - what activation function at each layer\n",
    "2. compile model\n",
    "    - specify loss functions\n",
    "    - details on optimization\n",
    "3. Fit\n",
    "    - cycle of backpropagation and optimization of model weights\n",
    "4. Predict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Step 1: Model specification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# read data\n",
    "# n_cols = number of nodes in input layer\n",
    "predictors = np.loadtext('predictors_data.csv', delimiter=',')\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# build model - Sequential is the easier of 2 ways to build a model\n",
    "# Sequential specifies weights only to 1 layer directly after it\n",
    "model = Sequential()\n",
    "# .add method adds layers\n",
    "# Called Dense b/c all nodes from previous layer connect to current layer\n",
    "# You may use layers that are not Dense\n",
    "# .Dense(number of layers, activation=activation function)\n",
    "# In first layer, you need to specify input_shape\n",
    "# Input_shape(n_cols columns, nothing here specifies any number of rows)\n",
    "# The last layer has 1 node b/c it's the output layer\n",
    "# This model has 2 hidden layers\n",
    "# Common to use 100s or 1000s of nodes in a layer\n",
    "model.add(Dense(100, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model specification - code without notes\n",
    "import numpy as np\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# read data\n",
    "predictors = np.loadtext('predictors_data.csv', delimiter=',')\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# build model\n",
    "model = Sequential()\n",
    "# add layers\n",
    "model.add(Dense(100, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: Understanding your data\n",
    "You will soon start building models in Keras to predict wages based on various professional and demographic factors. Before you start building a model, it's good to understand your data by performing some exploratory analysis.\n",
    "\n",
    "The data is pre-loaded into a pandas DataFrame called df. Use the .head() and .describe() methods in the IPython Shell for a quick overview of the DataFrame.\n",
    "\n",
    "The target variable you'll be predicting is wage_per_hour. Some of the predictor variables are binary indicators, where a value of 1 represents True, and 0 represents False.\n",
    "\n",
    "Of the 9 predictor variables in the DataFrame, how many are binary indicators? The min and max values as shown by .describe() will be informative here. How many binary indicator predictors are there?\n",
    "\n",
    "Answer: 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "In [1]: df.head()\n",
    "Out[1]: \n",
    "   wage_per_hour  union  education_yrs  experience_yrs  age  female  marr  \\\n",
    "0           5.10      0              8              21   35       1     1   \n",
    "1           4.95      0              9              42   57       1     1   \n",
    "2           6.67      0             12               1   19       0     0   \n",
    "3           4.00      0             12               4   22       0     0   \n",
    "4           7.50      0             12              17   35       0     1   \n",
    "\n",
    "   south  manufacturing  construction  \n",
    "0      0              1             0  \n",
    "1      0              1             0  \n",
    "2      0              1             0  \n",
    "3      0              0             0  \n",
    "4      0              0             0\n",
    "\n",
    "In [2]: df.describe()\n",
    "Out[2]: \n",
    "       wage_per_hour       union  education_yrs  experience_yrs         age  \\\n",
    "count     534.000000  534.000000     534.000000      534.000000  534.000000   \n",
    "mean        9.024064    0.179775      13.018727       17.822097   36.833333   \n",
    "std         5.139097    0.384360       2.615373       12.379710   11.726573   \n",
    "min         1.000000    0.000000       2.000000        0.000000   18.000000   \n",
    "25%         5.250000    0.000000      12.000000        8.000000   28.000000   \n",
    "50%         7.780000    0.000000      12.000000       15.000000   35.000000   \n",
    "75%        11.250000    0.000000      15.000000       26.000000   44.000000   \n",
    "max        44.500000    1.000000      18.000000       55.000000   64.000000   \n",
    "\n",
    "           female        marr       south  manufacturing  construction  \n",
    "count  534.000000  534.000000  534.000000     534.000000    534.000000  \n",
    "mean     0.458801    0.655431    0.292135       0.185393      0.044944  \n",
    "std      0.498767    0.475673    0.455170       0.388981      0.207375  \n",
    "min      0.000000    0.000000    0.000000       0.000000      0.000000  \n",
    "25%      0.000000    0.000000    0.000000       0.000000      0.000000  \n",
    "50%      0.000000    1.000000    0.000000       0.000000      0.000000  \n",
    "75%      1.000000    1.000000    1.000000       0.000000      0.000000  \n",
    "max      1.000000    1.000000    1.000000       1.000000      1.000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: Specifying a model\n",
    "Now you'll get to work with your first model in Keras, and will immediately be able to run more complex neural network models on larger datasets compared to the first two chapters.\n",
    "\n",
    "To start, you'll take the skeleton of a neural network and add a hidden layer and an output layer. You'll then fit that model and see Keras do the optimization so your model continually gets better.\n",
    "\n",
    "As a start, you'll predict workers wages based on characteristics like their industry, education and level of experience. You can find the dataset in a pandas dataframe called df. For convenience, everything in df except for the target has been converted to a NumPy matrix called predictors. The target, wage_per_hour, is available as a NumPy matrix called target.\n",
    "\n",
    "For all exercises in this chapter, we've imported the Sequential model constructor, the Dense layer constructor, and pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# Set up the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "# input_shape parameter to be the tuple (n_cols,) which means it has \n",
    "# n_cols items in each row of data, and any number of rows of data \n",
    "# are acceptable as inputs\n",
    "model.add(Dense(50, activation='relu', input_shape=(n_cols,)))\n",
    "\n",
    "# Add the second layer\n",
    "model.add(Dense(32, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Step 2 and 3: Compiling and fitting a model\n",
    "Why you need to compile a model - 2 args\n",
    "- Specify the optimizer\n",
    "    - Controls the learning rate\n",
    "    - many options and mathematically complex\n",
    "        - pragmatic solution: use a versatile algorithm for most problems\n",
    "        - \"Adam\" is usually a good choice as go-to optimizer\n",
    "            - adjusts learning rate as it does gradient descent to ensure reasonable values throughout weight optimization process\n",
    "- Loss function\n",
    "    - \"mean_squared_error\" common for regression\n",
    "    - use a different method for classification (see below)\n",
    "    \n",
    "Compiling a model - after specifying layers\n",
    "- model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "Fitting a model\n",
    "- applying backpropagation and gradient descent with your data to update the weights\n",
    "- similar to .fit() in sklearn but has more options\n",
    "- Note: Scaling data before fitting can ease optimization\n",
    "    - common approach: subtract each feature by feature mean and divide by std dev\n",
    "- model.fit(predictors, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compile the model\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# fit the model\n",
    "model.fit(predictors, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: Compiling the model\n",
    "You're now going to compile the model you specified earlier. To compile the model, you need to specify the optimizer and loss function to use. In the video, Dan mentioned that the Adam optimizer is an excellent choice. You can read more about it as well as other keras optimizers here (https://keras.io/optimizers/#adam), and if you are really curious to learn more, you can read the original paper that introduced the Adam optimizer (https://arxiv.org/abs/1412.6980v8).\n",
    "\n",
    "In this exercise, you'll use the Adam optimizer and the mean squared error loss function. Go for it! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Specify the model\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Verify that model contains information from compiling\n",
    "print(\"Loss function: \" + model.loss)\n",
    "# Loss function: mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: Fitting the model\n",
    "You're at the most fun part. You'll now fit the model. Recall that the data to be used as predictive features is loaded in a NumPy matrix called predictors and the data to be predicted is stored in a NumPy matrix called target. Your model is pre-written and it has been compiled with the code from the previous exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Specify the model\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors, target)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Classification models - with Deep Learning\n",
    "Classification - note biggest changes\n",
    "- loss function: 'categorical_crossentropy'\n",
    "    - instead of 'mean_squared_error' in above example\n",
    "    - there are other options, but it's the most common\n",
    "    - similar to log loss: Lower score is better\n",
    "    - add metrics=['accuracy'] to compile step for easy to understand diagnostics\n",
    "- Output layer has separate node for each possible outcomes and uses 'softmax' activation function\n",
    "    - 'softmax' ensures predictions sum to 1 so they can be interpreted as probabilities\n",
    "    \n",
    "Quick look at the data - binary classification problem\n",
    "- target (0 or 1): shot_result\n",
    "    - normally use separate column for each output - use keras function\n",
    "- Transforming to categorical using one-hot encoding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keras Classification\n",
    "from keras.utils import to_categorical\n",
    "data = pd.read_csv('basketball_shot_log.csv')\n",
    "predictors = data.drop(['shot_result'], axis=1).as_matrix()\n",
    "# one-hot encoding of target variables\n",
    "target = to_categorical(data.shot_results)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "# note: 2 nodes for 2 outcomes, and 'softmax' activation function\n",
    "model.add(Dense(2), activation = 'softmax')\n",
    "\n",
    "# Compile the model using categorical_crossentropy loss function\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.a  Understanding your classification data\n",
    "Now you will start modeling with a new dataset for a classification problem. This data includes information about passengers on the Titanic. You will use predictors such as age, fare and where each passenger embarked from to predict who will survive. This data is from a tutorial on data science competitions (https://www.kaggle.com/c/titanic). Look here (https://www.kaggle.com/c/titanic/data) for descriptions of the features.\n",
    "\n",
    "The data is pre-loaded in a pandas DataFrame called df.\n",
    "\n",
    "It's smart to review the maximum and minimum values of each variable to ensure the data isn't misformatted or corrupted. \n",
    "\n",
    "What was the maximum age of passengers on the Titanic? \n",
    "- Use the .describe() method in the IPython Shell to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "In [1]: df.describe()\n",
    "Out[1]: \n",
    "         survived      pclass         age       sibsp       parch        fare  \\\n",
    "count  891.000000  891.000000  891.000000  891.000000  891.000000  891.000000   \n",
    "mean     0.383838    2.308642   29.699118    0.523008    0.381594   32.204208   \n",
    "std      0.486592    0.836071   13.002015    1.102743    0.806057   49.693429   \n",
    "min      0.000000    1.000000    0.420000    0.000000    0.000000    0.000000   \n",
    "25%      0.000000    2.000000   22.000000    0.000000    0.000000    7.910400   \n",
    "50%      0.000000    3.000000   29.699118    0.000000    0.000000   14.454200   \n",
    "75%      1.000000    3.000000   35.000000    1.000000    0.000000   31.000000   \n",
    "max      1.000000    3.000000   80.000000    8.000000    6.000000  512.329200   \n",
    "\n",
    "             male  embarked_from_cherbourg  embarked_from_queenstown  \\\n",
    "count  891.000000               891.000000                891.000000   \n",
    "mean     0.647587                 0.188552                  0.086420   \n",
    "std      0.477990                 0.391372                  0.281141   \n",
    "min      0.000000                 0.000000                  0.000000   \n",
    "25%      0.000000                 0.000000                  0.000000   \n",
    "50%      1.000000                 0.000000                  0.000000   \n",
    "75%      1.000000                 0.000000                  0.000000   \n",
    "max      1.000000                 1.000000                  1.000000   \n",
    "\n",
    "       embarked_from_southampton  \n",
    "count                 891.000000  \n",
    "mean                    0.722783  \n",
    "std                     0.447876  \n",
    "min                     0.000000  \n",
    "25%                     0.000000  \n",
    "50%                     1.000000  \n",
    "75%                     1.000000  \n",
    "max                     1.000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: 80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.b  Last steps in classification models\n",
    "You'll now create a classification model using the titanic dataset, which has been pre-loaded into a DataFrame called df. You'll take information about the passengers and predict which ones survived.\n",
    "\n",
    "The predictive variables are stored in a NumPy array predictors. The target to predict is in df.survived, though you'll have to manipulate it for keras. The number of predictive features is stored in n_cols.\n",
    "\n",
    "Here, you'll use the 'sgd' optimizer, which stands for Stochastic Gradient Descent (https://en.wikipedia.org/wiki/Stochastic_gradient_descent). You'll learn more about this in the next chapter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "# Convert the target to categorical: target\n",
    "target = to_categorical(df.survived)\n",
    "\n",
    "# Set up the model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "model.add(Dense(32,activation='relu', input_shape=(n_cols,)))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='sgd',loss='categorical_crossentropy',\n",
    "metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors, target)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output\n",
    "Epoch 1/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 8.3484 - acc: 0.4062\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 2.5633 - acc: 0.5516\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 2.2398 - acc: 0.5859     \n",
    "Epoch 2/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 1.3865 - acc: 0.6250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.9423 - acc: 0.6196\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.9276 - acc: 0.6150     \n",
    "Epoch 3/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.9685 - acc: 0.6562\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6606 - acc: 0.6658\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6770 - acc: 0.6465     \n",
    "Epoch 4/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6190 - acc: 0.5938\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6970 - acc: 0.6549\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6720 - acc: 0.6712     \n",
    "Epoch 5/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.8023 - acc: 0.5625\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6204 - acc: 0.6739\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6173 - acc: 0.6756     \n",
    "Epoch 6/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6783 - acc: 0.5625\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6219 - acc: 0.6766\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6169 - acc: 0.6902     \n",
    "Epoch 7/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.4871 - acc: 0.8125\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6215 - acc: 0.6590\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6146 - acc: 0.6712     \n",
    "Epoch 8/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6327 - acc: 0.6250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.5996 - acc: 0.6970\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6023 - acc: 0.6902     \n",
    "Epoch 9/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.4249 - acc: 0.8438\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6173 - acc: 0.6861\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6157 - acc: 0.6835     \n",
    "Epoch 10/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.5451 - acc: 0.6875\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 0.6051 - acc: 0.6902\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6069 - acc: 0.6925\n",
    "Out[1]: <keras.callbacks.History at 0x7f88fc07f0f0>\n",
    "\n",
    "\n",
    "# This simple model is generating an accuracy of 68!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Using models\n",
    "- SAVE model after you train it\n",
    "- RELOAD model\n",
    "- Make PREDICTIONS with model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving, reloading, and using your model\n",
    "from keras.models import load_model\n",
    "model.save('model_file.h5')\n",
    "my_model = load_model('my_model.h5')\n",
    "predictions = my_model.predict(data_to_predict_with)\n",
    "# using NumPy indexing, only get probability that the show was made\n",
    "probability_true = predictions[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Verifying model structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Making predictions\n",
    "The trained network from your previous coding exercise is now stored as model. New data to make predictions is stored in a NumPy array as pred_data. Use model to make predictions on your new data.\n",
    "\n",
    "In this exercise, your predictions will be probabilities, which is the most common way for data scientists to communicate their predictions to colleagues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify, compile, and fit the model\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.compile(optimizer='sgd', \n",
    "              loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "model.fit(predictors, target)\n",
    "\n",
    "# Calculate predictions: predictions\n",
    "predictions = model.predict(pred_data)\n",
    "\n",
    "# Calculate predicted probability of survival: predicted_prob_true\n",
    "predicted_prob_true = predictions[:,1]\n",
    "\n",
    "# print predicted_prob_true\n",
    "print(predicted_prob_true)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output\n",
    "\n",
    "Epoch 1/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 9.4068 - acc: 0.3438\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 2.1078 - acc: 0.5924\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 2.0548 - acc: 0.6013     \n",
    "Epoch 2/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.9343 - acc: 0.5938\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.9755 - acc: 0.6372\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.9881 - acc: 0.6362     \n",
    "Epoch 3/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 3.1622 - acc: 0.2812\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.9660 - acc: 0.6576\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.9560 - acc: 0.6475     \n",
    "Epoch 4/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.7433 - acc: 0.7812\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.7070 - acc: 0.6739\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.7000 - acc: 0.6737     \n",
    "Epoch 5/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.7013 - acc: 0.7188\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6351 - acc: 0.6522\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6370 - acc: 0.6562     \n",
    "Epoch 6/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.7438 - acc: 0.5938\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6490 - acc: 0.6332\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6406 - acc: 0.6475     \n",
    "Epoch 7/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.5850 - acc: 0.6562\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6513 - acc: 0.6454\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6484 - acc: 0.6450     \n",
    "Epoch 8/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.6137 - acc: 0.6562\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6187 - acc: 0.6658\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6316 - acc: 0.6487     \n",
    "Epoch 9/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.6688 - acc: 0.6250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6231 - acc: 0.6576\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6204 - acc: 0.6588     \n",
    "Epoch 10/10\n",
    "\n",
    " 32/800 [>.............................] - ETA: 0s - loss: 0.6003 - acc: 0.6875\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/800 [==========================>...] - ETA: 0s - loss: 0.6177 - acc: 0.6753\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/800 [==============================] - 0s - loss: 0.6136 - acc: 0.6812     \n",
    "[0.21993461 0.3318482  0.24766211 0.51462483 0.16771315 0.14743942\n",
    " 0.07266742 0.2729067  0.17536648 0.4474381  0.18807179 0.25605577\n",
    " 0.1751357  0.34066525 0.15425213 0.11837941 0.22323096 0.4354514\n",
    " 0.07228678 0.42617893 0.42058256 0.19022378 0.07656103 0.2302739\n",
    " 0.2898574  0.17831704 0.4497322  0.39488176 0.18922023 0.44863054\n",
    " 0.4136607  0.39599928 0.15403347 0.21477048 0.27403453 0.44038844\n",
    " 0.2478449  0.16212454 0.44724074 0.4080968  0.2458698  0.36913165\n",
    " 0.41874805 0.11317883 0.29375517 0.08350059 0.40117604 0.12408093\n",
    " 0.4339563  0.5006711  0.318091   0.01506576 0.47888747 0.46149215\n",
    " 0.25830168 0.27560487 0.40160352 0.22523038 0.34403372 0.15403347\n",
    " 0.17595078 0.33919814 0.24056132 0.40442795 0.31600302 0.15790847\n",
    " 0.3180895  0.4518196  0.18083148 0.45688182 0.18818697 0.3890984\n",
    " 0.14759547 0.07269966 0.36255804 0.29960537 0.28081793 0.25597137\n",
    " 0.16022985 0.4513318  0.40332136 0.13509353 0.2791521  0.23960653\n",
    " 0.18508025 0.26095796 0.2958366  0.46594405 0.30380365 0.4570568\n",
    " 0.16814701]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Fine-tuning keras models\n",
    "Objective: How to choose model architecture and model optimization arguments\n",
    "\n",
    "Model optimization is hard\n",
    "- simultaneously optimizing 1000s of parameters of complex relationship9s\n",
    "- updates may not improve model meaningfully\n",
    "- updates too small (if learning rate is low) or too large (if learning rate is high)\n",
    "    - smart optimizer like adam works, but optimization problems can still occur \n",
    "\n",
    "### 4.1 Easiest way to see effect of different learning rates is using SGD\n",
    "- Stochastic Gradient Descent (SGD) - simplest optimizer\n",
    "    - learning rate = 0.01 is common\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stochastic Gradient Descent (SGD) - simplest optimizer\n",
    "def get_new_model(input_shape = input_shape):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(100, activation='relu', input_shape = input_shape))\n",
    "    model.add(Dense(100, activation='relu'))\n",
    "    model.add(Dense(2, activation='softmax'))\n",
    "    return(model)\n",
    "\n",
    "# test learning rates\n",
    "lr_to_test = [.000001, 0.01, 1]\n",
    "for lr in lr_to_test:\n",
    "    model = get_new_model()\n",
    "    my_optimizer = SGD(lr=lr)\n",
    "    model.compile(optimizer=my_optimizer, loss='categorical_crossentropy')\n",
    "    model.fit(predictors, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 The dying neuron problem\n",
    "- Even if your learning rate is well tuned, you can have the dying neuron problem.\n",
    "- Dying neuron problem =\n",
    "    - when a neuron takes a value < 0 (is negative) for all rows of data\n",
    "    - recall ReLU: negative input produces output 0 and slope 0 \n",
    "        - if slope 0, slopes of weights into the node are 0, so those weights don't get updated \n",
    "    - once a node starts always getting negative inputs\n",
    "        - it may continue only getting negative inputs\n",
    "    - contributes nothing to the model\n",
    "        - \"Dead\" neuron\n",
    "    - Attempted solutions\n",
    "        - using activation function with slope never 0\n",
    "            - Vanishing gradients problem - like tanh\n",
    "                - occurs when many layers have very small slopes (ie. due to being on flat part of tanh curve, on either end)\n",
    "                - may work in a model with a few hidden layers\n",
    "                - But in deep networks, updates to backprop were close to 0\n",
    "- If you are having problems, a potential solution may be changing the activation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Diagnosing optimization problems\n",
    "Which of the following could prevent a model from showing an improved loss in its first few epochs?\n",
    "\n",
    "Possible Answers:\n",
    "Learning rate too low.\n",
    "Learning rate too high.\n",
    "Poor choice of activation function.\n",
    "All of the above.\n",
    "\n",
    "Answer: All of the above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Changing optimization parameters\n",
    "It's time to get your hands dirty with optimization. You'll now try optimizing a model at a very low learning rate, a very high learning rate, and a \"just right\" learning rate. \n",
    "\n",
    "You'll want to look at the results after running this exercise, remembering that a LOW value for the LOSS FUNCTION is GOOD.\n",
    "\n",
    "For these exercises, we've pre-loaded the predictors and target values from your previous classification models (predicting who would survive on the Titanic). You'll want the optimization to start from scratch every time you change the learning rate, to give a fair comparison of how each learning rate did in your results. So we have created a function get_new_model() that creates an unoptimized model to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the SGD optimizer\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "# Create list of learning rates: lr_to_test\n",
    "lr_to_test = [.000001, 0.01, 1]\n",
    "\n",
    "# Loop over learning rates\n",
    "for lr in lr_to_test:\n",
    "    print('\\n\\nTesting model with learning rate: %f\\n'%lr )\n",
    "    \n",
    "    # Build new model to test, unaffected by previous models\n",
    "    model = get_new_model()\n",
    "    \n",
    "    # Create SGD optimizer with specified learning rate: my_optimizer\n",
    "    my_optimizer = SGD(lr=lr)\n",
    "    \n",
    "    # Compile the model\n",
    "    model.compile(optimizer=my_optimizer, loss='categorical_crossentropy')\n",
    "    \n",
    "    # Fit the model\n",
    "    model.fit(predictors, target)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Testing model with learning rate: 0.000001\n",
    "\n",
    "Epoch 1/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 1s - loss: 3.6053\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "416/891 [=============>................] - ETA: 0s - loss: 3.7014\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.6057     \n",
    "Epoch 2/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.5751\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "576/891 [==================>...........] - ETA: 0s - loss: 3.4996\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.5656     \n",
    "Epoch 3/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 2.6692\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "480/891 [===============>..............] - ETA: 0s - loss: 3.4125\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.5255     \n",
    "Epoch 4/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.0058\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "576/891 [==================>...........] - ETA: 0s - loss: 3.4704\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.4854     \n",
    "Epoch 5/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 2.5452\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "480/891 [===============>..............] - ETA: 0s - loss: 3.4640\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.4454     \n",
    "Epoch 6/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.4446\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "480/891 [===============>..............] - ETA: 0s - loss: 3.4553\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.4056     \n",
    "Epoch 7/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 4.1073\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "512/891 [================>.............] - ETA: 0s - loss: 3.4992\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.3659     \n",
    "Epoch 8/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.0972\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/891 [===================>..........] - ETA: 0s - loss: 3.2938\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.3263     \n",
    "Epoch 9/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.7464\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "544/891 [=================>............] - ETA: 0s - loss: 3.2872\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.2867     \n",
    "Epoch 10/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 3.3862\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/891 [===================>..........] - ETA: 0s - loss: 3.1217\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 3.2473     \n",
    "\n",
    "\n",
    "Testing model with learning rate: 0.010000\n",
    "\n",
    "Epoch 1/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 1s - loss: 1.0910\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "544/891 [=================>............] - ETA: 0s - loss: 1.8761\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 1.4069     \n",
    "Epoch 2/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 2.1146\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "576/891 [==================>...........] - ETA: 0s - loss: 0.7418\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.7022     \n",
    "Epoch 3/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.5700\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "640/891 [====================>.........] - ETA: 0s - loss: 0.6680\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6472     \n",
    "Epoch 4/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6192\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "640/891 [====================>.........] - ETA: 0s - loss: 0.6432\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6228     \n",
    "Epoch 5/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.4959\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "640/891 [====================>.........] - ETA: 0s - loss: 0.6158\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6199     \n",
    "Epoch 6/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6682\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "640/891 [====================>.........] - ETA: 0s - loss: 0.6064\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6010     \n",
    "Epoch 7/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6180\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "640/891 [====================>.........] - ETA: 0s - loss: 0.6076\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.5996     \n",
    "Epoch 8/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6059\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 0.5897\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.6055     \n",
    "Epoch 9/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6522\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 0.5991\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.5948     \n",
    "Epoch 10/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 0.6478\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 0.5789\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 0.5827     \n",
    "\n",
    "\n",
    "Testing model with learning rate: 1.000000\n",
    "\n",
    "Epoch 1/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 1s - loss: 1.0273\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "512/891 [================>.............] - ETA: 0s - loss: 5.4474\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 5.9885     \n",
    "Epoch 2/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 4.5332\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "544/891 [=================>............] - ETA: 0s - loss: 6.1628\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 3/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 7.0517\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "352/891 [==========>...................] - ETA: 0s - loss: 5.8611\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "736/891 [=======================>......] - ETA: 0s - loss: 6.2414\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 4/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 6.0443\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "416/891 [=============>................] - ETA: 0s - loss: 6.3155\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "800/891 [=========================>....] - ETA: 0s - loss: 6.1652\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 5/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 9.0664\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "384/891 [===========>..................] - ETA: 0s - loss: 6.5060\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 6/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 6.0443\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 6.2122\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 7/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 5.0369\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 6.2841\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 8/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 5.0369\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 6.1402\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 9/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 5.5406\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 6.0923\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867     \n",
    "Epoch 10/10\n",
    "\n",
    " 32/891 [>.............................] - ETA: 0s - loss: 5.5406\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "672/891 [=====================>........] - ETA: 0s - loss: 6.2362\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "891/891 [==============================] - 0s - loss: 6.1867"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Model Validation\n",
    "Validation in deep learning\n",
    "- commonly use validation split rather than cross validation\n",
    "- Deep learning widely used on large datasets\n",
    "    - so k-fold cross validation is not commonly used\n",
    "- Single validation score is based on large amount of data, and is reliable\n",
    "    - Repeated training from cross-validation would take long time\n",
    "\n",
    "Early Stopping - stop training when validation score is not improving\n",
    "\n",
    "Experimentation\n",
    "- experiment with different architectures\n",
    "- more layers\n",
    "- fewer layers\n",
    "- layers with more nodes\n",
    "- layers with fewer nodes\n",
    "- creating a great model requires experimentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5.a Model validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see :param accuracy metrics for a classification problem\n",
    "model.compile(optimizer='adam',loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "# see :param validation_split\n",
    "model,.fit(predictors, target, validation_split=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5.b Early Stopping - stop early if accuracy and \n",
    "- check output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import EarlyStopping\n",
    "# patience is number of epochs before stopping, 2 or 3 is enough\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "# note: callbacks param takes a list\n",
    "# Advanced: add several callbacks\n",
    "# keras Default: 10 epochs trained\n",
    "model.fit(predictors, target, validation_split=0.3, epochs=20,\n",
    "          callbacks=[early_stopping_monitor])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5.c Evaluating model accuracy on validation dataset\n",
    "Now it's your turn to monitor model accuracy with a validation data set. A model definition has been provided as model. Your job is to add the code to compile it and then fit it. You'll check the validation score in each epoch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "hist = model.fit(predictors, target, validation_split=0.3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train on 623 samples, validate on 268 samples\n",
    "Epoch 1/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 3.3028 - acc: 0.4062\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 1.3342 - acc: 0.5970\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 1.3117 - acc: 0.6035 - val_loss: 0.6835 - val_acc: 0.7201\n",
    "Epoch 2/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.6891 - acc: 0.7188\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.8822 - acc: 0.5714 - val_loss: 1.1022 - val_acc: 0.6418\n",
    "Epoch 3/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 1.0317 - acc: 0.5938\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.7970 - acc: 0.6228 - val_loss: 0.8587 - val_acc: 0.6343\n",
    "Epoch 4/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.6470 - acc: 0.6875\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.7579 - acc: 0.6497\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.7538 - acc: 0.6485 - val_loss: 0.6898 - val_acc: 0.7052\n",
    "Epoch 5/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.6848 - acc: 0.6562\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6774 - acc: 0.6453 - val_loss: 0.5892 - val_acc: 0.7164\n",
    "Epoch 6/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.5687 - acc: 0.6875\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "448/623 [====================>.........] - ETA: 0s - loss: 0.6688 - acc: 0.6518\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6599 - acc: 0.6517 - val_loss: 0.5295 - val_acc: 0.7500\n",
    "Epoch 7/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.5598 - acc: 0.7500\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "576/623 [==========================>...] - ETA: 0s - loss: 0.6072 - acc: 0.6771\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6013 - acc: 0.6806 - val_loss: 0.5124 - val_acc: 0.7164\n",
    "Epoch 8/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.5911 - acc: 0.7500\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.5914 - acc: 0.6908\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.5911 - acc: 0.6902 - val_loss: 0.5254 - val_acc: 0.7649\n",
    "Epoch 9/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.5627 - acc: 0.7500\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6727 - acc: 0.6597 - val_loss: 0.5689 - val_acc: 0.7052\n",
    "Epoch 10/10\n",
    "\n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.4831 - acc: 0.8125\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6196 - acc: 0.6886 - val_loss: 0.5382 - val_acc: 0.7425"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5.d Early stopping: Optimizing the optimization\n",
    "Now that you know how to monitor your model performance throughout optimization, you can use early stopping to stop optimization when it isn't helping any more. Since the optimization stops automatically when it isn't helping, you can also set a high value for epochs in your call to .fit(), as Dan showed in the video.\n",
    "\n",
    "The model you'll optimize has been specified as model. As before, the data is pre-loaded as predictors and target. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import EarlyStopping\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors, target, validation_split=0.3,\n",
    "          epochs=30, callbacks=[early_stopping_monitor])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train on 623 samples, validate on 268 samples\n",
    "    Epoch 1/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 5.6563 - acc: 0.4688\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 1.6479 - acc: 0.5625\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 1.6352 - acc: 0.5666 - val_loss: 1.0820 - val_acc: 0.6530\n",
    "    Epoch 2/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 1.8316 - acc: 0.4688\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.8358 - acc: 0.6086\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.8304 - acc: 0.6100 - val_loss: 0.5692 - val_acc: 0.7276\n",
    "    Epoch 3/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.8393 - acc: 0.6250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.7130 - acc: 0.6549 - val_loss: 0.5265 - val_acc: 0.7537\n",
    "    Epoch 4/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.9949 - acc: 0.6250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.6702 - acc: 0.6760\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6734 - acc: 0.6758 - val_loss: 0.5196 - val_acc: 0.7351\n",
    "    Epoch 5/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.5458 - acc: 0.7812\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.6727 - acc: 0.6480\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6808 - acc: 0.6469 - val_loss: 0.6424 - val_acc: 0.6903\n",
    "    Epoch 6/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.4596 - acc: 0.8438\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6279 - acc: 0.7127 - val_loss: 0.5721 - val_acc: 0.7239\n",
    "    Epoch 7/30\n",
    "    \n",
    " 32/623 [>.............................] - ETA: 0s - loss: 0.6590 - acc: 0.6562\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "608/623 [============================>.] - ETA: 0s - loss: 0.6533 - acc: 0.7007\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "623/623 [==============================] - 0s - loss: 0.6529 - acc: 0.6998 - val_loss: 0.6481 - val_acc: 0.6679"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6 Experimenting with wider networks\n",
    "Now you know everything you need to begin experimenting with different models!\n",
    "\n",
    "A model called model_1 has been pre-loaded. You can see a summary of this model printed in the IPython Shell. This is a relatively small network, with only 10 units in each hidden layer.\n",
    "\n",
    "In this exercise you'll create a new model called model_2 which is similar to model_1, except it has 100 units in each hidden layer.\n",
    "\n",
    "After you create model_2, both models will be fitted, and a graph showing both models loss score at each epoch will be shown. We added the argument verbose=False in the fitting commands to print out fewer updates, since you will look at these graphically instead of as text.\n",
    "\n",
    "Because you are fitting two models, it will take a moment to see the outputs after you hit run, so be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summary of model 1\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "dense_1 (Dense)              (None, 10)                110       \n",
    "_________________________________________________________________\n",
    "dense_2 (Dense)              (None, 10)                110       \n",
    "_________________________________________________________________\n",
    "dense_3 (Dense)              (None, 2)                 22        \n",
    "=================================================================\n",
    "Total params: 242.0\n",
    "Trainable params: 242\n",
    "Non-trainable params: 0.0\n",
    "_________________________________________________________________\n",
    "None\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create model 2 which has 100 units in each hidden layer\n",
    "\n",
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first and second layers\n",
    "model_2.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "model_2.add(Dense(100, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit model_1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model_2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', model_2_training.history['val_loss'], 'b')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model 2 has a lower loss value so is better than Model 1\n",
    "\n",
    "<img src=\"image8.png\" width=\"500\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.7 Adding layers to a network - Experiment with Deeper Network\n",
    "You've seen how to experiment with wider networks. In this exercise, you'll try a deeper network (more hidden layers).\n",
    "\n",
    "Once again, you have a baseline model called model_1 as a starting point. It has 1 hidden layer, with 50 units. You can see a summary of that model's structure printed out. You will create a similar network with 3 hidden layers (still keeping 50 units in each layer).\n",
    "\n",
    "This will again take a moment to fit both models, so you'll need to wait a few seconds to see the results after you run your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model 1\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "dense_1 (Dense)              (None, 50)                550       \n",
    "_________________________________________________________________\n",
    "dense_2 (Dense)              (None, 2)                 102       \n",
    "=================================================================\n",
    "Total params: 652.0\n",
    "Trainable params: 652\n",
    "Non-trainable params: 0.0\n",
    "_________________________________________________________________\n",
    "None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The input shape to use in the first hidden layer\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first, second, and third hidden layers\n",
    "model_2.add(Dense(50,activation='relu',input_shape=input_shape))\n",
    "model_2.add(Dense(50,activation='relu'))\n",
    "model_2.add(Dense(50,activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam',\n",
    "                loss='categorical_crossentropy',\n",
    "                metrics=['accuracy'])\n",
    "\n",
    "# Fit model 1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model 2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', model_2_training.history['val_loss'], 'b')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both models, you should look for the best val_loss and val_acc, which won't be the last epoch for that model.\n",
    "\n",
    "The blue model is Model 2 and the red is the original Model 1. The model with the lower loss value is the better model.\n",
    "\n",
    "<img src=\"image9.png\" width=\"500\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.8 Thinking about Model Capacity (or Network Capacity)\n",
    "- Takes some practive to get intuition to know what experiments and architectures to try.\n",
    "    - Thinking about Model Capacity can guide us\n",
    "\n",
    "Model Capacity\n",
    "- this term is closely associated with Overfitting and Underfitting\n",
    "- Model Capacity = model's ability to capture predictive patterns in your data\n",
    "    - more capacity >> towards Low Bias and High Variance\n",
    "    - Increase capacity by:\n",
    "        - Increase nodes/neurons in hidden layer\n",
    "        - Increase layers\n",
    "\n",
    "Workflow for optimizing model capacity\n",
    "- Start with a small network\n",
    "- Get the validation score\n",
    "- Keep increasing capacity until validation score is no longer improving\n",
    "- Example: Sequential experiments - increased nodes, then layers, adjust\n",
    "    - Hidden layer 1, Nodes/layer 100, MSE 5.4, Next step: Increase capacity\n",
    "    - Hidden layer 1, Nodes/layer 250, MSE 4.8, Next step: Increase capacity\n",
    "    - Hidden layer 2, Nodes/layer 250, MSE 4.4, Next step: Increase capacity\n",
    "    - Hidden layer 3, Nodes/layer 250, MSE 4.5, Next step: Decrease capacity\n",
    "    - Hidden layer 3, Nodes/layer 200, MSE 4.3, Next step: Done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.9 Experimenting with model structures\n",
    "You've just run an experiment where you compared two networks that were identical except that the 2nd network had an extra hidden layer. You see that this 2nd network (the deeper network) had better performance. Given that, which of the following would be a good experiment to run next for even better performance?\n",
    "\n",
    "Possible Answers:\n",
    "- Try a new network with fewer layers than anything you have tried yet.\n",
    "- Use more units in each hidden layer.\n",
    "- Use fewer units in each hidden layer.\n",
    "\n",
    "Answer: \n",
    "Use more units in each hidden layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.10 Stepping up to Images\n",
    "- MNIST dataset\n",
    "    - images of handwritten digits\n",
    "    - 28x28 grid flattened to 784 values for each image (784x1 array)\n",
    "    - value in each part of array denotes darkness of that pixel\n",
    "        - 0 is darkest\n",
    "        - 255 is lightest\n",
    "    - digits 0-9\n",
    "- deep learning model to predict the digit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.11 Building your own digit recognition model\n",
    "You've reached the final exercise of the course - you now know everything you need to build an accurate model to recognize handwritten digits!\n",
    "\n",
    "We've already done the basic manipulation of the MNIST dataset shown in the video, so you have X and y loaded and ready to model with. Sequential and Dense from keras are also pre-imported.\n",
    "\n",
    "To add an extra challenge, we've loaded only 2500 images, rather than 60000 which you will see in some published results. Deep learning models perform better with more data, however, they also take longer to train, especially when they start becoming more complex.\n",
    "\n",
    "If you have a computer with a CUDA compatible GPU, you can take advantage of it to improve computation time. If you don't have a GPU, no problem! You can set up a deep learning environment in the cloud that can run your models on a GPU. Here is a blog post by Dan that explains how to do this - check it out after completing this exercise! https://www.datacamp.com/community/tutorials/deep-learning-jupyter-aws\n",
    "\n",
    "It is a great next step as you continue your deep learning journey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first hidden layer\n",
    "model.add(Dense(50, activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Add the second hidden layer\n",
    "model.add(Dense(50, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X, y, validation_split=0.3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    Train on 1750 samples, validate on 750 samples\n",
    "    Epoch 1/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 3s - loss: 2.3008 - acc: 0.1250\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 320/1750 [====>.........................] - ETA: 0s - loss: 2.1986 - acc: 0.2125\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 672/1750 [==========>...................] - ETA: 0s - loss: 2.0420 - acc: 0.3229\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1056/1750 [=================>............] - ETA: 0s - loss: 1.8622 - acc: 0.4167\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1344/1750 [======================>.......] - ETA: 0s - loss: 1.7421 - acc: 0.4680\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1696/1750 [============================>.] - ETA: 0s - loss: 1.6015 - acc: 0.5195\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 1.5804 - acc: 0.5280 - val_loss: 0.8387 - val_acc: 0.7933\n",
    "    Epoch 2/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.7431 - acc: 0.9375\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 480/1750 [=======>......................] - ETA: 0s - loss: 0.7692 - acc: 0.8187\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 864/1750 [=============>................] - ETA: 0s - loss: 0.6975 - acc: 0.8241\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1152/1750 [==================>...........] - ETA: 0s - loss: 0.6736 - acc: 0.8247\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1280/1750 [====================>.........] - ETA: 0s - loss: 0.6563 - acc: 0.8289\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1472/1750 [========================>.....] - ETA: 0s - loss: 0.6378 - acc: 0.8308\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1632/1750 [==========================>...] - ETA: 0s - loss: 0.6333 - acc: 0.8333\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.6268 - acc: 0.8354 - val_loss: 0.5000 - val_acc: 0.8720\n",
    "    Epoch 3/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.6002 - acc: 0.7812\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 448/1750 [======>.......................] - ETA: 0s - loss: 0.4587 - acc: 0.8750\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 928/1750 [==============>...............] - ETA: 0s - loss: 0.4409 - acc: 0.8858\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1408/1750 [=======================>......] - ETA: 0s - loss: 0.4191 - acc: 0.8878\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.4096 - acc: 0.8903 - val_loss: 0.4262 - val_acc: 0.8693\n",
    "    Epoch 4/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.4248 - acc: 0.8438\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 512/1750 [=======>......................] - ETA: 0s - loss: 0.3218 - acc: 0.9180\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 864/1750 [=============>................] - ETA: 0s - loss: 0.3151 - acc: 0.9190\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1152/1750 [==================>...........] - ETA: 0s - loss: 0.3197 - acc: 0.9097\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1440/1750 [=======================>......] - ETA: 0s - loss: 0.3124 - acc: 0.9125\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1728/1750 [============================>.] - ETA: 0s - loss: 0.3146 - acc: 0.9120\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.3176 - acc: 0.9109 - val_loss: 0.3806 - val_acc: 0.8733\n",
    "    Epoch 5/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.1267 - acc: 1.0000\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 512/1750 [=======>......................] - ETA: 0s - loss: 0.2501 - acc: 0.9414\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 992/1750 [================>.............] - ETA: 0s - loss: 0.2473 - acc: 0.9365\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1472/1750 [========================>.....] - ETA: 0s - loss: 0.2582 - acc: 0.9334\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.2529 - acc: 0.9354 - val_loss: 0.3595 - val_acc: 0.8947\n",
    "    Epoch 6/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.1885 - acc: 0.9688\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 512/1750 [=======>......................] - ETA: 0s - loss: 0.2542 - acc: 0.9395\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 992/1750 [================>.............] - ETA: 0s - loss: 0.2131 - acc: 0.9526\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1312/1750 [=====================>........] - ETA: 0s - loss: 0.2069 - acc: 0.9520\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1696/1750 [============================>.] - ETA: 0s - loss: 0.2126 - acc: 0.9475\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.2126 - acc: 0.9474 - val_loss: 0.3391 - val_acc: 0.8920\n",
    "    Epoch 7/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.2002 - acc: 0.9688\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 480/1750 [=======>......................] - ETA: 0s - loss: 0.1501 - acc: 0.9750\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 864/1750 [=============>................] - ETA: 0s - loss: 0.1551 - acc: 0.9676\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1280/1750 [====================>.........] - ETA: 0s - loss: 0.1728 - acc: 0.9563\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1632/1750 [==========================>...] - ETA: 0s - loss: 0.1744 - acc: 0.9559\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.1744 - acc: 0.9554 - val_loss: 0.3268 - val_acc: 0.8960\n",
    "    Epoch 8/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.2347 - acc: 0.9688\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 416/1750 [======>.......................] - ETA: 0s - loss: 0.1399 - acc: 0.9712\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 800/1750 [============>.................] - ETA: 0s - loss: 0.1431 - acc: 0.9725\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1216/1750 [===================>..........] - ETA: 0s - loss: 0.1594 - acc: 0.9663\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1664/1750 [===========================>..] - ETA: 0s - loss: 0.1540 - acc: 0.9669\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.1509 - acc: 0.9674 - val_loss: 0.3195 - val_acc: 0.8960\n",
    "    Epoch 9/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.1214 - acc: 0.9375\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 512/1750 [=======>......................] - ETA: 0s - loss: 0.1125 - acc: 0.9766\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 992/1750 [================>.............] - ETA: 0s - loss: 0.1264 - acc: 0.9738\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1440/1750 [=======================>......] - ETA: 0s - loss: 0.1226 - acc: 0.9771\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.1191 - acc: 0.9777 - val_loss: 0.3337 - val_acc: 0.9040\n",
    "    Epoch 10/10\n",
    "    \n",
    "  32/1750 [..............................] - ETA: 0s - loss: 0.1376 - acc: 1.0000\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 512/1750 [=======>......................] - ETA: 0s - loss: 0.1001 - acc: 0.9844\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    " 992/1750 [================>.............] - ETA: 0s - loss: 0.1058 - acc: 0.9819\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1472/1750 [========================>.....] - ETA: 0s - loss: 0.1043 - acc: 0.9810\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
    "1750/1750 [==============================] - 0s - loss: 0.1015 - acc: 0.9806 - val_loss: 0.3343 - val_acc: 0.9013"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see better than 90% accuracy recognizing handwritten digits, even while using a small training set of only 1750 images!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.12 Final Thoughts\n",
    "Next steps\n",
    "- Start with standard prediction problems on tables of numbers\n",
    "- Images (with convolutional neural networks) are common next steps\n",
    "- data types: text, sound, etc\n",
    "- Kaggle has great datasets and forums to learn\n",
    "- Check out wikipedia page: List of datasets for machine learning research\n",
    "- keras.io for excellent documentation\n",
    "- keras and tensorflow Github repos are good\n",
    "- Graphical processing unit (GPU) provides dramatic speedups in model training times\n",
    "    - need a CUDA compatible GPU\n",
    "    - most NVIDIA GPUs are compatible\n",
    "- For training on using GPUs in the cloud look here:\n",
    "    - http://bit.ly/2mYQXQb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}