P01-C05-autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic differentiation with ``autograd`` \n",
    "\n",
    "\n",
    "In machine learning, we *train* models to get better and better as a function of experience. Usually, *getting better* means minimizing a *loss function*, i.e. a score that answers \"how *bad* is our model?\" With neural networks, we choose loss functions to be differentiable with respect to our parameters.\n",
    "\n",
    "Put simply, this means that for each of the model's parameters, we can determine how much *increasing* or *decreasing* it might affect the loss. While the calculations are straightforward, for complex models, working it out by hand can be a pain.\n",
    "\n",
    "_MXNet_'s autograd package expedites this work by automatically calculating derivatives. And while most other libraries require that we compile a symbolic graph to take automatic derivatives, ``mxnet.autograd``, like PyTorch, allows you to take derivatives while writing  ordinary imperative code.\n",
    "\n",
    "Every time you make pass through your model, ``autograd`` builds a graph on the fly, through which it can immediately backpropagate gradients.\n",
    "\n",
    "Let's go through it step by step. For this tutorial, we'll only need to import ``mxnet.ndarray``, and ``mxnet.autograd``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import mxnet.ndarray as nd\n",
    "import mxnet.autograd as ag\n",
    "mx.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attaching gradients\n",
    "\n",
    "As a toy example, Let's say that we are interested in differentiating a function ``f = 2 * (x ** 2)`` with respect to parameter x. We can start by assigning an initial value of ``x``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = nd.array([[1, 2], [3, 4]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we compute the gradient of ``f`` with respect to ``x``, we'll need a place to store it. In _MXNet_, we can tell an NDArray that we plan to store a gradient by invoking its ``attach_grad()`` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x.attach_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're going to define out function ``f`` and *MXNet* will generate a computation graph on the fly. It's as if *MXNet* turned on a recording device and captured the exact path by which each variable was generated. \n",
    "\n",
    "Note that building the computation graph requires a nontrivial amount of computation. So we only *MXNet* to build the graph when explicitly told to do so. We can instruct *MXNet* to start recording by placing code inside a ``with autograd.record():`` block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with ag.record():\n",
    "  y = x * 2\n",
    "  z = y * x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's backprop by calling ``z.backward()``. When ``z`` has more than one entry, ``z.backward() is equivalent to mx.nd.sum(z).backward().\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see if this is the expected output. Remember that ``y = x * 2``, and ``z = x * y``, so ``z`` should be equal to  ``2 * x * x``. After, doing backprop with ``z.backward()``, we expect to get back gradient dz/dx as follows: dy/dx = ``2``, dz/dx = ``4 * x``. So, if everything went according to plan, ``x.grad`` should consist of an NDArray with the values ``[[4, 8],[12, 16]]``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[  4.   8.]\n",
      " [ 12.  16.]]\n",
      "<NDArray 2x2 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Head gradients and the chain rule\n",
    "\n",
    "*Warning: This part is tricky, but not necessary to understanding subsequent sections.*\n",
    "\n",
    "Sometimes when we call the backward method on an NDArray, e.g. ``y.backward()``, where ``y`` is a function of ``x`` we are just interested in the derivative of ``y`` with respect to ``x``. At other times, we may be interested in the gradient of ``z`` with respect to ``x``, where ``z`` is a function of ``y``. Recall that by the chain rule dz/dx can be expressed in terms of dz/dy and dy/dx. So, when ``y`` is part of a larger function ``z``, and we want ``x.grad`` to store dz/dx, we can pass in the *head gradient* dz/dy as an input to ``backward()``. The default argument is ``nd.ones_like(y)``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[ 40.           8.        ]\n",
      " [  1.20000005   0.16      ]]\n",
      "<NDArray 2x2 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "with ag.record():\n",
    "  y = x * 2\n",
    "  z = y * x\n",
    "    \n",
    "head_gradient = nd.array([[10,1.],[.1,.01]])\n",
    "z.backward(head_gradient)\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know the basics, we can do some wild things with autograd, including building diferentiable functions using Pythonic control flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = nd.random_normal(shape=3)\n",
    "a.attach_grad()\n",
    "\n",
    "with ag.record():\n",
    "    b = a * 2\n",
    "    while (nd.norm(b) < 1000).asscalar():\n",
    "        b = b * 2\n",
    "\n",
    "    if (mx.nd.sum(b) > 0).asscalar():\n",
    "        c = b\n",
    "    else :\n",
    "        c = 100 * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "head_gradient = nd.array([0.01,1.0,.1])\n",
    "c.backward(head_gradient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[   2048.  204800.   20480.]\n",
      "<NDArray 3 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "print(a.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}