chapter02_supervised-learning/section4-multiclass-logistic-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Julia Things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Environment\n",
    "\n",
    "First things first. Let us set up the environment with the requried packages for this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OS: Linux\n",
      "Julia: 0.6.0\n",
      "Knet: 0.8.5+\n",
      "GPU: NVS 310\n",
      "TITAN X (Pascal)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for p in (\"Knet\", \"Plots\", \"Plotly.jl\")\n",
    "    Pkg.installed(p) == nothing && Pkg.add(p)\n",
    "end\n",
    "\n",
    "using Knet, Plots\n",
    "gr()\n",
    "\n",
    "Knet.gpu(0); # set the desired GPU to use\n",
    "atype = Array{Float32}; # atype = KnetArray{Float32} for gpu usage, Array{Float32} for cpu. \n",
    "\n",
    "println(\"OS: \", Sys.KERNEL)\n",
    "println(\"Julia: \", VERSION)\n",
    "println(\"Knet: \", Pkg.installed(\"Knet\"))\n",
    "println(\"GPU: \", readstring(`nvidia-smi --query-gpu=name --format=csv,noheader`))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### New Stuff\n",
    "\n",
    "In this notebook we introduce the following Julia/Knet packages and functions:\n",
    "\n",
    "* Knet's function [accuracy](http://denizyuret.github.io/Knet.jl/latest/reference.html#Knet.accuracy): Knet's function `accuracy` allows you to automatically calculate the accuracy of your model. For example, as long as the function `predict` has been defined, and the training/testing data has been processed through Julia's function [minibatch](http://denizyuret.github.io/Knet.jl/latest/reference.html#Knet.minibatch) such that `dtrn=minibatch(xtrn,ytrn)`, `accuracy` allows you to calculate the accuracy as `acc=accuracy(w,dtrn,predict)`, where `w` is your model (i.e the parameters/weights of your model). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Multiclass Logistic Regression \n",
    "\n",
    "If you've made it through our tutorials on linear and logistic regression, then you're past the hardest part. You already know how to load and manipulate data, build computation graphs on the fly, and take derivatives. You also know how to define a loss function, construct a model, and write your own optimizer. Nearly all neural networks that we'll build in the real world consist of these same fundamental parts. The main differences will be the type and scale of the data and the complexity of the models. And every year or two, a new hipster optimizer comes around, but at their core they're all subtle variations of stochastic gradient descent.\n",
    "\n",
    "In [the previous chapter](../chapter02_supervised-learning/logistic-regression.ipynb), we introduced logistic regression, a classic algorithm for performing binary classification. We implemented a model \n",
    "\n",
    "$$\\hat{y} = \\sigma(\\boldsymbol{w}^T \\boldsymbol{x} + b)$$\n",
    "where $\\sigma$ is the sigmoid squashing function. This activation function on the final layer was crucial because it forced our outputs to take values in the range [0,1]. That allowed us to interpret these outputs as probabilties.\n",
    "We then updated our parameters to give the true labels (which take values either 1 or 0) the highest probability.\n",
    "In that tutorial, we looked at predicting whether or not an individual's income exceeded $50k based on features available in 1994 census data, and we also worked with the medical appoinment data. \n",
    "\n",
    "Binary classification is quite useful. We can use it to predict spam vs. not spam or cancer vs not cancer. \n",
    "But not every problem fits the mold of binary classification. Sometimes we encounter a problem where each example could belong to one of $k$ classes. For example, a photograph might depict a cat or a dog or a zebra or ... (you get the point). Given $K$ classes, the most naive way to solve a *multiclass classification* problem \n",
    "is to train $K$ different binary classifiers $f_k(\\boldsymbol{x})$. We could then predict that an example $\\boldsymbol{x}$ belongs  to the class $k$ for which the probability that the label applies is highest $\\max_{k} $f_k$(\\boldsymbol{x})$.\n",
    "\n",
    "There's a smarter way to go about this. We could force the output layer to be a discrete probability distribution over the $K$ classes. To be valid probabity distribution, we'll want the output $\\hat{y}$ to (i) contain only non-negative values, and (ii) sum to 1. We accomplish this by using the *softmax* function. Given an input vector $z$, softmax does two things. First, it exponentiates (elementwise) $e^{z}$, forcing all values to be strictly positive.\n",
    "Then it normalizes so that all values sum to $1$. Following the softmax operation computes the following\n",
    "\n",
    "\n",
    "$$\\mbox{softmax}(\\boldsymbol{z}) = \\frac{e^{\\boldsymbol{z}} }{\\sum_{k=1}^K e^{z_i}}$$\n",
    "\n",
    "Because now we have $K$ outputs and not $1$ we'll need weights connecting each of our inputs to each of our outputs. Graphically, the network looks something like this:\n",
    "\n",
    "![](../img/simple-softmax-net.png)\n",
    "\n",
    "We can represent these weights one for each input node, output node pair in a matrix $W$. We generate the linear mapping from inputs to outputs via a matrix-vector product $W \\boldsymbol{x} + \\boldsymbol{b}$. Note that the bias term is now a vector, with one component for each output node. The whole model, including the ativation function can be written:\n",
    "\n",
    "$$\\hat{y} = \\mbox{softmax}(W\\boldsymbol{x} + \\boldsymbol{b})$$\n",
    "\n",
    "This model is sometimes called *multiclass logistic regression*. \n",
    "Other common names for it include *softmax regression* and *multinomial regression*.\n",
    "For these concepts to sink in, let's actually implement softmax regression,\n",
    "and pick a slightly more interesting dataset this time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST dataset\n",
    "\n",
    "In this example we build a simple classification model for the [MNIST](http://yann.lecun.com/exdb/mnist/) handwritten digit recognition dataset. MNIST has 60000 training and 10000 test examples. Each input x consists of 784 pixels representing a 28x28 image. The corresponding output indicates the identity of the digit 0..9.\n",
    "\n",
    "![png](https://jamesmccaffrey.files.wordpress.com/2014/06/firsteightimages.jpg) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we'll use Knet's utilities for grabbing a copy of this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[36mINFO: \u001b[39m\u001b[22m\u001b[36mLoading MNIST...\n",
      "\u001b[39m"
     ]
    }
   ],
   "source": [
    "include(Knet.dir(\"data\",\"mnist.jl\"));\n",
    "xtrn, ytrn, xtst, ytst = mnist();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two parts of the dataset for training and testing. Each part has N items and each item is a tuple of an image and a label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((28, 28, 1, 60000), (60000,))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "size(xtrn), size(ytrn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that each image has been formatted as a 3-tuple (height, width, channel). For color images, the channel would have 3 dimensions (red, green and blue). In this case we have a gray scale image with a single channel. Note that `ytrn` is a `Array{UInt8,1}` data type. Let us take a look at these labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(UInt8[0x05, 0x0a, 0x04, 0x01, 0x09, 0x02, 0x01, 0x03, 0x01, 0x04], [5, 10, 4, 1, 9, 2, 1, 3, 1, 4])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ytrn[1:10], 1ytrn[1:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let us now use Knet's minibatch function to prepare the date for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtrn = minibatch(xtrn, ytrn, 100; xtype=atype);\n",
    "dtst = minibatch(xtst, ytst, 100; xtype=atype);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the input images have been reshaped to 784x1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Array{Float32,N} where N, Array{UInt8,1})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtrn.xtype, dtrn.ytype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass logistic regression\n",
    "\n",
    "Recall from our previous topic on [logistic regression](logistic-regression.ipynb) that for the loss function we can use the *negative log probability:*\n",
    "\n",
    "$$ l(\\theta) = - \\sum_i^n\\log\\big(\\mathbb{P}(y_i|\\boldsymbol{x}_i)\\big)$$\n",
    "\n",
    "where $y_i\\in\\mathbb{R}$ and $\\boldsymbol{x}_i\\in\\mathbb{R}^d$. For binary logistic regression, where we deal with only two classes, we've seen that we can write \n",
    "\n",
    "$$\\mathbb{P}(y_i|\\hat{y}_i) = \\sigma(\\hat{y}_i)^{y_i}(1-\\sigma(\\hat{y}_i))^{1-y_i}$$\n",
    "\n",
    "where $\\hat{y}_i$ is the probability that  $x_i$ has a given label. In that case we found the loss function to be\n",
    "\n",
    "$$ l = \\sum_{i=1}^n y\\log\\hat{y}_i + (1-y)\\log(1-\\hat{y}_i) $$\n",
    "\n",
    "This equation is the negative log likelihood of the Bernoulli distribution. Now, consider the multiclass case where we apply the softmax function instead of the logistic function:\n",
    "\n",
    "$$\\hat{y}_i = \\frac{e^{z_i}}{\\sum_{j=1}^K e^{z_j}}$$\n",
    "\n",
    "For binary classification problems, the softmax function outputs two values (between 0 and 1 and sum to 1) to give the prediction of each class. While the sigmoid function outputs one value (between 0 and 1) to give the prediction of one class (so the other class is 1-p). See [here](https://stats.stackexchange.com/questions/198038/cross-entropy-or-log-likelihood-in-output-layer) for an example. \n",
    "\n",
    "Before we can start training, we're going to need to define a loss function that makes sense when our prediction is a  probability distribution. \n",
    "\n",
    "The relevant loss function here is called [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy). The cross entropy between two probability distributions $p$ and $q$ is defined as\n",
    "\n",
    "$$H(p,q)=\\operatorname {E}_{p}[-\\log q]=H(p)+D_{{{\\mathrm  {KL}}}}(p\\|q),\\!$$\n",
    "\n",
    "where $H(p,q)$ is the entropy of $p$, and $D_{{{\\mathrm  {KL}}}}(p\\|q)$ is the  is the Kullback–Leibler divergence of $q$ from $p$. For discrete $p$ and $q$ this means \n",
    "\n",
    "$$H(p,q)=\\operatorname {E}_{p}[-\\log q]=-\\sum_{i=1}^n p_i \\log q_i$$\n",
    "\n",
    "Note that this is simply the expected value of $-\\log q$ given the probability distribution $p$. For example, for binary labels such that $p=\\{0,1\\}$ and $q=\\{\\hat{y}, 1-\\hat{y}\\}$ we can write\n",
    "\n",
    "$$H(p,q)=-\\sum_{i=1}^n y_i \\log  \\hat{y} + (1-y_i) \\log (1-\\hat{y})$$\n",
    "\n",
    "which is exactly the log loss for binary labels. For more information on this topic see [here](https://jamesmccaffrey.wordpress.com/2016/09/25/log-loss-and-cross-entropy-are-almost-the-same/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation\n",
    "\n",
    "Assument that $y$ is an $K\\times m$ matrix, where $m$ is the number of samples and $K$ is the number of labels. A Naive implementaion of this theory is a follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cross_entropy (generic function with 1 method)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax(z) = exp.(z) ./ sum(exp.(z), 1)\n",
    "cross_entropy(yhat, y) = - sum(y .* log.(yhat), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m = 1000;\n",
    "K = 10;\n",
    "z = rand(K, m);\n",
    "yhat = softmax(z);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's confirm that indeed all of our rows sum to 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "\u001b[91mUndefVarError: p not defined\u001b[39m",
     "output_type": "error",
     "traceback": [
      "\u001b[91mUndefVarError: p not defined\u001b[39m",
      "",
      "Stacktrace:",
      " [1] \u001b[1minclude_string\u001b[22m\u001b[22m\u001b[1m(\u001b[22m\u001b[22m::String, ::String\u001b[1m)\u001b[22m\u001b[22m at \u001b[1m./loading.jl:515\u001b[22m\u001b[22m"
     ]
    }
   ],
   "source": [
    "mapslices(indmax, p, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could then implement the loss functions as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cross_entropy (generic function with 1 method)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(yhat, y) = - sum(sum(z .* log.(z), 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, `log.` computes the logarithm of each element of `yhat`. Next, we multiply each element of `y` with the corresponding element of `log(yhat)`. `sum( , 1)` adds the elements in the first dimension of y (to understand this, notice that for binary labels we have to sum $y_i \\log  \\hat{y} + (1-y_i) \\log (1-\\hat{y})$ before we sum over the samples). Finally, the last sume computes the sum over all the examples in the batch (sometimes this is a mean instead of a sum). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Note that in the [source code](https://github.com/denizyuret/Knet.jl/blob/786a3cde09c8699801798af2b89de6dc552ed9be/src/loss.jl#L157-L167), we don't use this formulation, because it is numerically unstable. In Julia, `nll` computes the negative log likelihood of your predictions compared to the correct answers. Roughly, `nll` does something like this:\n",
    "\n",
    "$$\\log(\\hat{y}_i) = z_i - \\log \\big(\\sum_{j=1}^K e^{z_j}\\big)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see how `nll` works. First we create a random linear prediction `z`, and then we run it through `softmax`. The labels are also ramdomized. Do notice tha `nll` do not require one-hot labels. Instead, we simply create an array with random labels from 1 to $K$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = Array{UInt8,2}(rand(1:K, 1, m));\n",
    "z = rand(K, m);\n",
    "yhat = softmax(rand(K, m));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((Array{UInt8,2}, (1, 1000)), (Array{Float64,2}, (10, 1000)), [10, 10])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(typeof(y), size(y)), (typeof(yhat), size(yhat)), (y[1, 1:2] * 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now let us make sure `nll` gives an expected output. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.3016897033631283"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nll(yhat, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this number what we expected? Consider that our original predictions were random chosen. Since there are 10 labels, on average we would be correct 1/10th of the time. The:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.3025850929940455"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-log(1/10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. It's time to build our model and run our tests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(::gradfun) (generic function with 1 method)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(w,x) = w[1]*mat(x) .+ w[2]\n",
    "loss(w,x,ygold) = nll(predict(w,x), ygold)\n",
    "lossgradient = grad(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we assume ygold is an array of N integers indicating the correct answers for N instances (we use ygold=10 to represent the 0 answer) and predict() gives us a (10,N) matrix of scores for each answer. [mat](http://denizyuret.github.io/Knet.jl/latest/reference.html#Knet.mat) is needed to convert the (28,28,1,N) x array to a (784,N) matrix so it can be used in matrix multiplication. Other than the change of loss function, the softmax model is identical to the linear regression model. We use the same predict (except for mat reshaping), train and set lossgradient=grad(loss) as before.\n",
    "\n",
    "Now let's train a model on the MNIST data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "train (generic function with 1 method)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function train(w, data; lr=.1)\n",
    "    for (x,y) in data\n",
    "        dw = lossgradient(w, x, y)\n",
    "        for i in 1:length(w)\n",
    "            w[i] -= lr * dw[i]\n",
    "        end   \n",
    "    end\n",
    "\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(:epoch, 0, :trn, 0.10478333333333334, :tst, 0.1111)\n",
      "(:epoch, 1, :trn, 0.9003333333333333, :tst, 0.9049)\n",
      "(:epoch, 2, :trn, 0.9081166666666667, :tst, 0.9096)\n",
      "(:epoch, 3, :trn, 0.91195, :tst, 0.9111)\n",
      "(:epoch, 4, :trn, 0.9139833333333334, :tst, 0.9121)\n",
      "(:epoch, 5, :trn, 0.91535, :tst, 0.9137)\n",
      "(:epoch, 6, :trn, 0.9167166666666666, :tst, 0.9149)\n",
      "(:epoch, 7, :trn, 0.9176, :tst, 0.9145)\n",
      "(:epoch, 8, :trn, 0.9185166666666666, :tst, 0.9156)\n",
      "(:epoch, 9, :trn, 0.9194166666666667, :tst, 0.9154)\n",
      "(:epoch, 10, :trn, 0.9196833333333333, :tst, 0.9155)\n"
     ]
    }
   ],
   "source": [
    "w = map(atype, Any[ 0.1f0*randn(Float32, 10, 784), zeros(Float32, 10, 1) ])\n",
    "w = Any[ 0.1f0*randn(Float32,10,784), zeros(Float32,10,1) ];\n",
    "println((:epoch, 0, :trn, accuracy(w,dtrn,predict), :tst, accuracy(w,dtst,predict)))\n",
    "for epoch=1:10\n",
    "    train(w, dtrn; lr=0.5)\n",
    "    println((:epoch, epoch, :trn, accuracy(w,dtrn,predict), :tst, accuracy(w,dtst,predict)))\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let us check the accuracy on the testing and training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9196833333333333, 0.9155)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(w, dtrn, predict), accuracy(w, dtst, predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jeepers. We can get more than 90% accuracy on both data sets at this task just by training a linear model for a few seconds! You might reasonably conclude that this problem is too easy to be taken seriously by experts.\n",
    "But until recently, many papers (Google Scholar says 13,800) were published using results obtained on this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next:\n",
    "\n",
    "[Regularization](section5-regularization.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "For whinges or inquiries, open an issue on [GitHub](https://github.com/moralesq/Knet-the-Julia-dope)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.0",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}