chapter01_crashcourse/KnetArray.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Manipulate data the Knet way with `KnetArray`\n",
    "\n",
    "It's impossible to get anything done if we can't manipulate data. \n",
    "Generally, there are two important things we need to do with: \n",
    "(i) acquire it! and (ii) process it once it's inside the computer.\n",
    "There's no point in trying to acquire data if we don't even know how to store it,\n",
    "so let's get our hands dirty first by playing with synthetic data.\n",
    "\n",
    "We'll start by introducing KnetArrays, Knet's primary tool for storing and transforming data with GPUs. Although Knet can use Julia's `Array` type for standard CPU computations, GPUs have become indispensable for training large deep learning models. Even the small examples implemented here run up to 17x faster on the GPU compared to the 8 core CPU architecture we use for benchmarking. However GPU implementations have a few potential pitfalls: (i) GPU memory allocation is slow, (ii) GPU-RAM memory transfer is slow, (iii) reduction operations (like sum) can be very slow unless implemented properly ([See Optimizing Parallel Reduction in CUDA](http://developer.download.nvidia.com/compute/cuda/1.1-Beta/x86_website/projects/reduction/doc/reduction.pdf)).\n",
    "Knet implements [KnetArray](http://denizyuret.github.io/Knet.jl/latest/reference.html#KnetArray-1) as a Julia data type that wraps GPU array pointers. KnetArray is based on the more standard [CudaArray](https://github.com/JuliaGPU/CUDArt.jl) with a few important differences: (i) Garbage collection: KnetArrays have a custom memory manager, similar to [ArrayFire](https://arrayfire.com/), which reuse pointers garbage collected by Julia to reduce the number of GPU memory allocations, (ii) Slicing: contiguous array ranges (e.g. a[:,3:5]) are handled as views with shared pointers instead of copies when possible, and (iii) Broadcasting: a number of custom CUDA kernels written for KnetArrays implement element-wise, broadcasting, and scalar and vector reduction operations efficiently. As a result Knet allows users to implement their models using high-level code, yet be competitive in performance with other frameworks.\n",
    "\n",
    "\n",
    "## Getting started\n",
    "\n",
    "In this chapter, we'll get you going with the basic functionality. Don't worry if you don't understand any of the basic math, like element-wise operations or normal distributions. In the next two chapters we'll take another pass at KnetArray, teaching you both the math you'll need and how to realize it in code.\n",
    "\n",
    "Before we get started with Knet and KnetArrays, you should be aware of two special types of arrays: matrices and vectors. In particular, note that for any type T:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array{T,2} where T"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Array{T,2} where T \n",
    "b = Matrix{T} where T\n",
    "a == b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Array{T,1} where T \n",
    "b = Vector{T} where T\n",
    "a == b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to get started, let's import `Knet`. We’ll make a habit of setting a random seed with `srand` so that you always get the same results that we do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "using Knet\n",
    "srand(1);\n",
    "Knet.gpu(0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see how to create either KnetArray for gpu, without any values initialized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " 1.0  1.0  1.0   2.88255 \n",
       " 1.0  1.0  1.0  -0.295943\n",
       " 1.0  1.0  1.0  -1.86218 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = KnetArray{Float64}(3, 4)\n",
    "display(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Array(type, dims...) simply returns an uninitialized dense array without setting the values of any of its entries. This means that the entries can have any form of values, including very big ones! But typically, we'll want our arrays initialized. Commonly, we want all zeros:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×5 Knet.KnetArray{Float64,2}:\n",
       " 0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = KnetArray(zeros(3, 5))\n",
    "display(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simmilarty, we can create an array of all ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×5 Knet.KnetArray{Float64,2}:\n",
       " 1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = KnetArray(ones(3, 5))\n",
    "display(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, we'll want to create arrays whose values are sampled randomly. This is especially common when we intend to use the array as a parameter in a neural network. In this snippet, we initialize a 2-element array with values drawn from a standard normal distribution with zero mean and unit variance, one element with shape 3x4 and the other with shape 1x1 (i.e. bias). Julia's `map` applies a function to each value of an array and returns a new array containing the resulting values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Knet.KnetArray{Float64,2},1}:\n",
       " Knet.KnetArray{Float64,2}(Knet.KnetPtr(Ptr{Void} @0x000001020d600600, 96, 0, nothing), (3, 4))\n",
       " Knet.KnetArray{Float64,2}(Knet.KnetPtr(Ptr{Void} @0x000001020d600800, 8, 0, nothing), (1, 1)) "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = map(KnetArray, [randn(3, 4), randn(1,1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       "  0.297288  -0.0104452   2.29509   0.431422\n",
       "  0.382396  -0.839027   -2.26709   0.583708\n",
       " -0.597634   0.311111    0.529966  0.963272"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "1×1 Knet.KnetArray{Float64,2}:\n",
       " 0.458791"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x[1]), display(x[2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pointer of a KnetArray is accessible via the `.ptr` attribte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Knet.KnetPtr(Ptr{Void} @0x000001020d600600, 96, 0, nothing), Knet.KnetPtr(Ptr{Void} @0x000001020d600800, 8, 0, nothing))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1].ptr, x[2].ptr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dimensions of each KnetArray are accessible via the `.dims` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 4), (1, 1))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1].dims, x[2].dims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also query its `length`, which is equal to the product of the components of the shape. Together with the precision of the stored values, this tells us how much memory the array occupies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length(x[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations\n",
    "\n",
    "KnetArray supports a large number of standard mathematical operations. Such as element-wise addition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = 1:10;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr x: Ptr{Void} @0x00007f61175757f0\n",
      "UnitRange{Int64}\n",
      "  start: Int64 1\n",
      "  stop: Int64 10\n"
     ]
    }
   ],
   "source": [
    "print(\"ptr x: \", pointer_from_objref(x), \"\\n\")\n",
    "dump(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr x: Ptr{Void} @0x00007f6116eaa8f0\n",
      "Base.ReshapedArray{Int64,2,UnitRange{Int64},Tuple{}}\n",
      "  parent: UnitRange{Int64}\n",
      "    start: Int64 1\n",
      "    stop: Int64 10\n",
      "  dims: Tuple{Int64,Int64}\n",
      "    1: Int64 2\n",
      "    2: Int64 5\n",
      "  mi: Tuple{} ()\n"
     ]
    }
   ],
   "source": [
    "x = reshape(x, 2,5);\n",
    "print(\"ptr x: \", pointer_from_objref(x), \"\\n\")\n",
    "dump(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10-element Array{Int64,1}:\n",
       "  1\n",
       "  2\n",
       "  3\n",
       "  4\n",
       "  5\n",
       "  6\n",
       "  7\n",
       "  8\n",
       "  9\n",
       " 10"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collect(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Knet.KnetArray{Int64,1}(Knet.KnetPtr(Ptr{Void} @0x000001020d600e00, 80, 0, nothing), (10,))"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = KnetArray(collect(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knet.KnetArray{Int64,1}\n",
      "  ptr: Knet.KnetPtr\n",
      "    ptr: Ptr{Void} Ptr{Void} @0x000001020d600c00\n",
      "    len: Int64 80\n",
      "    dev: Int64 0\n",
      "    parent: Void nothing\n",
      "  dims: Tuple{Int64}\n",
      "    1: Int64 10\n"
     ]
    }
   ],
   "source": [
    "dump(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knet.KnetArray{Int64,1}\n",
      "  ptr: Knet.KnetPtr\n",
      "    ptr: Ptr{Void} Ptr{Void} @0x000001020d600e00\n",
      "    len: Int64 80\n",
      "    dev: Int64 0\n",
      "    parent: Void nothing\n",
      "  dims: Tuple{Int64}\n",
      "    1: Int64 10\n"
     ]
    }
   ],
   "source": [
    "dump(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "using MappedArrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Array{Int64,2}:\n",
       " 1  4  7  10\n",
       " 2  5  8  11\n",
       " 3  6  9  12"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M = reshape([1:12;], 3, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 MappedArrays.ReadonlyMappedArray{Float64,2,Array{Int64,2},Base.#sqrt}:\n",
       " 1.0      2.0      2.64575  3.16228\n",
       " 1.41421  2.23607  2.82843  3.31662\n",
       " 1.73205  2.44949  3.0      3.4641 "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M2 = mappedarray(√, M) # no floating points are stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MappedArrays.ReadonlyMappedArray{Float64,2,Array{Int64,2},Base.#sqrt}\n",
      "  f: sqrt (function of type Base.#sqrt)\n",
      "  data: Array{Int64}((3, 4)) [1 4 7 10; 2 5 8 11; 3 6 9 12]\n"
     ]
    }
   ],
   "source": [
    "dump(M2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Knet.KnetArray{Float64,2}(Knet.KnetPtr(Ptr{Void} @0x000001020d600000, 96, 0, nothing), (3, 4))"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M2gpu = KnetArray(M2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " 1.0      2.0      2.64575  3.16228\n",
       " 1.41421  2.23607  2.82843  3.31662\n",
       " 1.73205  2.44949  3.0      3.4641 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(M2gpu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UnitRange{Int64}\n",
      "  start: Int64 1\n",
      "  stop: Int64 2\n",
      "UnitRange{Int64}\n",
      "  start: Int64 1\n",
      "  stop: Int64 12\n"
     ]
    }
   ],
   "source": [
    "x = 1:2;\n",
    "y = 1:12;\n",
    "dump(x)\n",
    "dump(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knet.KnetArray{Int64,1}\n",
      "  ptr: Knet.KnetPtr\n",
      "    ptr: Ptr{Void} Ptr{Void} @0x000001020d601000\n",
      "    len: Int64 16\n",
      "    dev: Int64 0\n",
      "    parent: Void nothing\n",
      "  dims: Tuple{Int64}\n",
      "    1: Int64 2\n",
      "Knet.KnetArray{Int64,1}\n",
      "  ptr: Knet.KnetPtr\n",
      "    ptr: Ptr{Void} Ptr{Void} @0x000001020d601200\n",
      "    len: Int64 96\n",
      "    dev: Int64 0\n",
      "    parent: Void nothing\n",
      "  dims: Tuple{Int64}\n",
      "    1: Int64 12\n"
     ]
    }
   ],
   "source": [
    "xx = KnetArray(x)\n",
    "yy = KnetArray(y)\n",
    "dump(xx)\n",
    "dump(yy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knet.KnetArray{Float64,2}\n",
      "  ptr: Knet.KnetPtr\n",
      "    ptr: Ptr{Void} Ptr{Void} @0x000001020d600000\n",
      "    len: Int64 96\n",
      "    dev: Int64 0\n",
      "    parent: Void nothing\n",
      "  dims: Tuple{Int64,Int64}\n",
      "    1: Int64 3\n",
      "    2: Int64 4\n"
     ]
    }
   ],
   "source": [
    "dump(M2gpu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 MappedArrays.MappedArray{Float64,2,Array{Int64,2},Base.#sqrt,#square}:\n",
       " 1.0      2.0      2.64575  3.16228\n",
       " 1.41421  2.23607  2.82843  3.31662\n",
       " 1.73205  2.44949  3.0      3.4641 "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "square(x) = x^2\n",
    "\n",
    "M3 = mappedarray((√, square), M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = KnetArray(randn(3, 4));\n",
    "y = KnetArray(randn(3, 4));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " -2.31208     0.143274   -3.45468   2.88255 \n",
       " -0.144817   -1.75026    -1.06113  -0.295943\n",
       "  0.0786531  -0.0403134  -1.56271  -1.86218 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x .+ y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       "  1.29937     -0.467873   2.90532    1.88425 \n",
       " -1.71794      0.0520587  0.280033  -1.72752 \n",
       " -0.00747342  -0.115824   0.474897   0.710209"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x .* y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " 0.259621  2.13697   0.134351  6.55758 \n",
       " 3.45662   0.970187  0.566171  0.229782\n",
       " 0.945873  0.696921  0.316755  0.585546"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(exp.(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also grab a matrix's transpose to compute a proper matrix-matrix product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×3 Knet.KnetArray{Float64,2}:\n",
       "  5.62106   3.75899   -1.60386\n",
       " -1.82654  -3.11336    2.34333\n",
       "  1.40378   0.635362   1.06181"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x * y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll explain these operations and present even more operators in the [linear algebra](P01-C03-linear-algebra.ipynb) chapter. But for now, we'll stick with the mechanics of working with Arrays/KnetArrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (i) garbage collection & In-place operations\n",
    "\n",
    "In the previous example, every time we ran an operation, we allocated new memory to host its results. For example, if we write `y = x + y`, we will dereference the matrix that `y` used to point to and instead point it at the newly allocated memory. In the following example we demonstrate this with the `.ptr` attribute, which gives us the exact address of the referenced object in memory. After running `y = y + x`, we'll find that `y.ptr` points to a different location. That's because Knet first evaluates `y + x`, allocating new memory for the result and then subsequently redirects `y` to point at this new location in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020d601e00, 96, 0, nothing)\n",
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020d602000, 96, 0, nothing)\n"
     ]
    }
   ],
   "source": [
    "print(\"ptr y: \", y.ptr, \"\\n\")\n",
    "y = y + x\n",
    "print(\"ptr y: \", y.ptr, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This might be undesirable for two reasons. First, we don't want to run around allocating memory unnecessarily all the time. In machine learning, we might have hundreds of megabytes of paramaters and update all of them multiple times per second. Typically, we'll want to perform these updates in place. Second, we might point at the same parameters from multiple variables. If we don't update in place, this could cause a memory leak, and could cause us to inadvertently reference stale parameters.\n",
    "Fortunately, performing in-place operations in MXNet is easy. We can assign the result of an operation to a previously allocated array with slice notation, e.g., y[:] = < expression >."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020d602000, 96, 0, nothing)\n",
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020d602000, 96, 0, nothing)\n"
     ]
    }
   ],
   "source": [
    "print(\"ptr y: \", y.ptr, \"\\n\")\n",
    "y[:] = y + x\n",
    "print(\"ptr y: \", y.ptr, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knet models do not overwrite arrays which need to be preserved for\n",
    "gradient calculation.  This leads to a lot of allocation and regular\n",
    "GPU memory allocation is prohibitively slow. Fortunately most models\n",
    "use identically sized arrays over and over again, so we can minimize\n",
    "the number of actual allocations by reusing preallocated but garbage\n",
    "collected pointers.\n",
    "When Julia gc reclaims a KnetArray, a special finalizer keeps its\n",
    "pointer in a table instead of releasing the memory.  If an array with\n",
    "the same size in bytes is later requested, the same pointer is reused.\n",
    "The exact algorithm for allocation is:\n",
    "1. Try to find a previously allocated and garbage collected pointer in\n",
    "   the current device. (0.5 μs)\n",
    "2. If not available, try to allocate a new array using cudaMalloc. (10\n",
    "   μs)\n",
    "3. If not successful, try running gc() and see if we get a pointer of\n",
    "   the right size. (75 ms, but this should be amortized over all\n",
    "   reusable pointers that become available due to the gc)\n",
    "4. Finally if all else fails, clean up all saved pointers in the\n",
    "   current device using cudaFree and try allocation one last\n",
    "   time. (25-70 ms, however this causes the elimination of all\n",
    "   reusable pointers)\n",
    "   \n",
    "For example, if we create new arrays x and y, Knet will try to find a previously allocated and garbace collected pointer in the current device (in this case GPU 0). Since the dimensions are larger, it will allocate a new array using cudaMalloc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = KnetArray(randn(9, 16));\n",
    "y = KnetArray(randn(9, 16));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, notice that the pointer of y changes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr x: Knet.KnetPtr(Ptr{Void} @0x000001020dc00000, 1152, 0, nothing)\n",
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020dc00600, 1152, 0, nothing)\n",
      "ptr x: Knet.KnetPtr(Ptr{Void} @0x000001020dc00000, 1152, 0, nothing)\n",
      "ptr y: Knet.KnetPtr(Ptr{Void} @0x000001020dc00c00, 1152, 0, nothing)\n"
     ]
    }
   ],
   "source": [
    "print(\"ptr x: \", x.ptr, \"\\n\")\n",
    "print(\"ptr y: \", y.ptr, \"\\n\")\n",
    "y = y + x\n",
    "print(\"ptr x: \", x.ptr, \"\\n\")\n",
    "print(\"ptr y: \", y.ptr, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can manually run garbage collection with `gc()` to ensure the unsued pointer is collected. Then, Knet will find this pointer and ensure it has the same size as the one requested for z. Note below that indeed this is the same pointer that was previously assigned to z. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptr z: Knet.KnetPtr(Ptr{Void} @0x000001020dc00600, 1152, 0, nothing)\n"
     ]
    }
   ],
   "source": [
    "gc()\n",
    "z = KnetArray(randn(9, 16));\n",
    "print(\"ptr z: \", z.ptr, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (ii) Slicing\n",
    "Knet KnetArrays support slicing in all the ridiculous ways you might imagine accessing your data. Here's an example of reading the second and third rows from `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Knet.KnetArray{Float64,1}:\n",
       " -1.56058 \n",
       " -1.39425 \n",
       "  0.131417"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = KnetArray(randn(3, 4));\n",
    "display(x[1:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try writing to a specific element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1,2] = 9.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " -1.56058    9.0        -1.27235   -0.613233 \n",
       " -1.39425   -0.0608202   1.94879    0.0166144\n",
       "  0.131417   0.746144   -0.562271   2.08423  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multi-dimensional slicing is also supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2×3 Knet.KnetArray{Float64,2}:\n",
       " -1.56058   9.0        -1.27235\n",
       " -1.39425  -0.0608202   1.94879"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x[1:2, 1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1:2, 1:3] = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×4 Knet.KnetArray{Float64,2}:\n",
       " 5.0       5.0        5.0       -0.613233 \n",
       " 5.0       5.0        5.0        0.0166144\n",
       " 0.131417  0.746144  -0.562271   2.08423  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (iii) Broadcasting\n",
    "\n",
    "You might wonder, what happens if you add a vector `y` to a matrix `X`? These operations, where we compose a low dimensional array `y` with a high-dimensional array `X` invoke a functionality called [broadcasting](https://docs.julialang.org/en/stable/manual/functions/#man-vectorized-1). Here, the low-dimensional array is duplicated along any axis with dimension $1$ to match the shape of the high dimensional array. Broadcasting operators supported include: (.*), (.+), (.-), (./), (.<), (.<=), (.!=), (.==), (.>), (.>=), (.^), max, min. (Boolean operators generate outputs with same type as inputs; no support for KnetArray{Bool}). Consider the following example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×3 Knet.KnetArray{Float64,2}:\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3-element Knet.KnetArray{Float64,1}:\n",
       " 0.0\n",
       " 1.0\n",
       " 2.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = KnetArray(ones(3,3))\n",
    "y = KnetArray(0:2.);\n",
    "display(x), display(y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×3 Knet.KnetArray{Float64,2}:\n",
       " 1.0  1.0  1.0\n",
       " 2.0  2.0  2.0\n",
       " 3.0  3.0  3.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(x  .+ y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting from KnetArray to Array\n",
    "Converting KnetArray to and from Array is easy. The converted arrays do not share memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×3 Array{Float64,2}:\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = Array(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×3 Knet.KnetArray{Float64,2}:\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0\n",
       " 1.0  1.0  1.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = KnetArray(b)\n",
    "display(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "[Linear algebra](../chapter01_crashcourse/linear-algebra.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/moralesq/Knet-the-Julia-dope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.0-pre.alpha",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}