P03-C01-mlp-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilayer perceptrons from scratch\n",
    "\n",
    "Now that we've covered all the preliminaries, extending to deep neural networks is actually quite easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd\n",
    "import numpy as np\n",
    "ctx = mx.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST data (surprise!)\n",
    "\n",
    "Let's go ahead and grab our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnist = mx.test_utils.get_mnist()\n",
    "batch_size = 64\n",
    "train_data = mx.io.NDArrayIter(mnist[\"train_data\"], mnist[\"train_label\"], batch_size, shuffle=True)\n",
    "test_data = mx.io.NDArrayIter(mnist[\"test_data\"], mnist[\"test_label\"], batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer perceptrons\n",
    "\n",
    "Here's where things start to get interesting. Before, we mapped our inputs directly onto our outputs through a single linear transformation. \n",
    "\n",
    "![](https://github.com/zackchase/mxnet-the-straight-dope/blob/master/img/simple-softmax-net.png?raw=true)\n",
    "\n",
    "This model is perfectly adequate when the underlying relationship between our data points and labels is approximately linear. When our data points and targets are characterized by a more complex relationship, a linear model and produce sucky results. We can model a more general class of functions by incorporating one or more *hidden layers*.\n",
    "\n",
    "![](https://github.com/zackchase/mxnet-the-straight-dope/blob/master/img/multilayer-perceptron.png?raw=true)\n",
    "\n",
    "Here, each layer will require it's own set of parameters. To make things simple here, we'll assume two hidden layers of computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W1 = nd.random_normal(shape=(784,256)) *.01\n",
    "b1 = nd.random_normal(shape=256) * .01\n",
    "\n",
    "W2 = nd.random_normal(shape=(256,128)) *.01\n",
    "b2 = nd.random_normal(shape=128) * .01\n",
    "\n",
    "W3 = nd.random_normal(shape=(128,10)) *.01\n",
    "b3 = nd.random_normal(shape=10) *.01\n",
    "\n",
    "params = [W1, b1, W2, b2, W3, b3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's allocate space for gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for param in params:\n",
    "    param.attach_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation functions\n",
    "\n",
    "If we compose a multi-layer network but use only linear operations, then our entire network will still be a linear function. That's because $\\hat{y} = X \\cdot W_1 \\cdot W_2 \\cdot W_2 = X \\cdot W_4 $ for $W_4 = W_1 \\cdot W_2 \\cdot W3$. To give our model the capacity to capture nonlinear functions, we'll need to interleave our linear operations with activation functions. In this case, we'll use the rectified linear unit (ReLU):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def relu(X):\n",
    "    return nd.maximum(X,nd.zeros_like(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax output\n",
    "\n",
    "As with multiclass logistic regression, we'll want out outputs to be *stochastic*, meaning that they constitute a valid probability distribution. We'll use the same softmax activation functino on our output to make sure that our outputs sum to one and are non-negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(y_linear):\n",
    "    exp = nd.exp(y_linear-nd.max(y_linear))\n",
    "    partition =nd.sum(exp, axis=0, exclude=True).reshape((-1,1))\n",
    "    return exp / partition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model\n",
    "\n",
    "Now we're ready to define our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    #######################\n",
    "    #  Compute the first hidden layer \n",
    "    #######################\n",
    "    h1_linear = nd.dot(X, W1) + b1\n",
    "    h1 = relu(h1_linear)\n",
    "    \n",
    "    #######################\n",
    "    #  Compute the second hidden layer\n",
    "    #######################\n",
    "    h2_linear = nd.dot(h1, W2) + b2\n",
    "    h2 = relu(h2_linear)\n",
    "    \n",
    "    #######################\n",
    "    #  Compute the output layer\n",
    "    #######################\n",
    "    yhat_linear = nd.dot(h2, W3) + b3\n",
    "    yhat = softmax(yhat_linear)\n",
    "    \n",
    "    return yhat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The cross-entropy loss function\n",
    "\n",
    "Nothing changes here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cross_entropy(yhat, y):\n",
    "    return - nd.sum(y * nd.log(yhat), axis=0, exclude=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def SGD(params, lr):    \n",
    "    for param in params:\n",
    "        param[:] = param - lr * param.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iterator, net):\n",
    "    numerator = 0.\n",
    "    denominator = 0.\n",
    "    data_iterator.reset()\n",
    "    for i, batch in enumerate(data_iterator):\n",
    "        data = batch.data[0].as_in_context(ctx).reshape((-1,784))\n",
    "        label = batch.label[0].as_in_context(ctx)\n",
    "        label_one_hot = nd.one_hot(label, 10)\n",
    "        output = net(data)\n",
    "        predictions = nd.argmax(output, axis=1)\n",
    "        numerator += nd.sum(predictions == label)\n",
    "        denominator += data.shape[0]\n",
    "    return (numerator / denominator).asscalar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute the training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0. Loss: 0.172049476541, Train_acc 0.955674, Test_acc 0.953125\n",
      "Epoch 1. Loss: 0.104753504359, Train_acc 0.966518, Test_acc 0.962082\n",
      "Epoch 2. Loss: 0.0736007969344, Train_acc 0.977912, Test_acc 0.968153\n",
      "Epoch 3. Loss: 0.0588375955563, Train_acc 0.978045, Test_acc 0.96467\n",
      "Epoch 4. Loss: 0.0483494877147, Train_acc 0.986574, Test_acc 0.970641\n"
     ]
    }
   ],
   "source": [
    "epochs = 5\n",
    "moving_loss = 0.\n",
    "\n",
    "for e in range(epochs):\n",
    "    train_data.reset()\n",
    "    for i, batch in enumerate(train_data):\n",
    "        data = batch.data[0].as_in_context(ctx).reshape((-1,784))\n",
    "        label = batch.label[0].as_in_context(ctx)\n",
    "        label_one_hot = nd.one_hot(label, 10)\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            loss = cross_entropy(output, label_one_hot)\n",
    "        loss.backward()\n",
    "        SGD(params, .01)\n",
    "        \n",
    "        ##########################\n",
    "        #  Keep a moving average of the losses\n",
    "        ##########################\n",
    "        if i == 0:\n",
    "            moving_loss = nd.mean(loss).asscalar()\n",
    "        else:\n",
    "            moving_loss = .99 * moving_loss + .01 * nd.mean(loss).asscalar()\n",
    "            \n",
    "    test_accuracy = evaluate_accuracy(test_data, net)\n",
    "    train_accuracy = evaluate_accuracy(train_data, net)\n",
    "    print(\"Epoch %s. Loss: %s, Train_acc %s, Test_acc %s\" % (e, moving_loss, train_accuracy, test_accuracy)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Noice. With just two hidden layers containing 256 and 128 hidden nodes, repsectively, we can achieve over 95% accuracy on this task. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}