PyMC3_Multiple_Computational_Backends.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\" style=\"margin-top: 1em;\"><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Scope-of-the-Project\" data-toc-modified-id=\"Scope-of-the-Project-0.1\">Scope of the Project</a></span></li><li><span><a href=\"#Changes-to-the-PyMC3-Code-Base\" data-toc-modified-id=\"Changes-to-the-PyMC3-Code-Base-0.2\">Changes to the PyMC3 Code Base</a></span><ul class=\"toc-item\"><li><span><a href=\"#A-new-module:-'Backends_Symbolic'\" data-toc-modified-id=\"A-new-module:-'Backends_Symbolic'-0.2.1\">A new module: 'Backends_Symbolic'</a></span></li><li><span><a href=\"#Changes-needed-for-model.py\" data-toc-modified-id=\"Changes-needed-for-model.py-0.2.2\">Changes needed for model.py</a></span><ul class=\"toc-item\"><li><span><a href=\"#TensorVariables\" data-toc-modified-id=\"TensorVariables-0.2.2.1\">TensorVariables</a></span></li><li><span><a href=\"#TensorVariable-related-functions\" data-toc-modified-id=\"TensorVariable-related-functions-0.2.2.2\">TensorVariable-related functions</a></span></li></ul></li><li><span><a href=\"#Changes-needed-for-distribution.py\" data-toc-modified-id=\"Changes-needed-for-distribution.py-0.2.3\">Changes needed for distribution.py</a></span><ul class=\"toc-item\"><li><span><a href=\"#Getting-Values-from-Symbolic-Variables\" data-toc-modified-id=\"Getting-Values-from-Symbolic-Variables-0.2.3.1\">Getting Values from Symbolic Variables</a></span></li><li><span><a href=\"#TensorFlow-Sessions\" data-toc-modified-id=\"TensorFlow-Sessions-0.2.3.2\">TensorFlow Sessions</a></span></li><li><span><a href=\"#Functions\" data-toc-modified-id=\"Functions-0.2.3.3\">Functions</a></span></li></ul></li><li><span><a href=\"#Changes-needed-for-continuous.py\" data-toc-modified-id=\"Changes-needed-for-continuous.py-0.2.4\">Changes needed for continuous.py</a></span><ul class=\"toc-item\"><li><span><a href=\"#Basic-Math\" data-toc-modified-id=\"Basic-Math-0.2.4.1\">Basic Math</a></span></li></ul></li><li><span><a href=\"#Changes-needed-for-arraystep.py\" data-toc-modified-id=\"Changes-needed-for-arraystep.py-0.2.5\">Changes needed for arraystep.py</a></span></li><li><span><a href=\"#Changes-needed-for-metropolis.py\" data-toc-modified-id=\"Changes-needed-for-metropolis.py-0.2.6\">Changes needed for metropolis.py</a></span></li></ul></li><li><span><a href=\"#Test-Case:-A-Motivating-Example:-Linear-Regression\" data-toc-modified-id=\"Test-Case:-A-Motivating-Example:-Linear-Regression-0.3\">Test Case: A Motivating Example: Linear Regression</a></span><ul class=\"toc-item\"><li><span><a href=\"#Running-with-original-pymc3-code-(restart-kernel)\" data-toc-modified-id=\"Running-with-original-pymc3-code-(restart-kernel)-0.3.1\">Running with original pymc3 code (restart kernel)</a></span></li><li><span><a href=\"#Running-with-Theano-backend-(restart-kernel-for-a-fresh-import)\" data-toc-modified-id=\"Running-with-Theano-backend-(restart-kernel-for-a-fresh-import)-0.3.2\">Running with Theano backend (restart kernel for a fresh import)</a></span></li><li><span><a href=\"#Running-with-TensorFlow-backend-(restart-kernel-for-a-fresh-import)\" data-toc-modified-id=\"Running-with-TensorFlow-backend-(restart-kernel-for-a-fresh-import)-0.3.3\">Running with TensorFlow backend (restart kernel for a fresh import)</a></span></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scope of the Project\n",
    "\n",
    "PyMC3 does much of its computation using [Theano](http://deeplearning.net/software/theano/). However, the developers of PyMC3 have expressed interest in allowing for multiple computational backends for pymc3. This project has been started as PyMC4 (see this [discussion thread](https://discourse.pymc.io/t/tensorflow-backend-for-pymc4/409/15)).\n",
    "\n",
    "Within this github repo, I altered some of the PyMC3 source code so that it can run on either Theano or [TensorFlow](https://www.tensorflow.org/) backends. [Keras](https://keras.io/) is a package with similar functionality, and I used it as a model for this project (see backends [documenation](https://keras.io/backend/) or the [source code](https://github.com/keras-team/keras/tree/master/keras/backend) for Keras).\n",
    "\n",
    "Note that the term 'computational backends' (or 'symbolic backends') differs from 'backends' in PyMC3, which is used for different storage options (hdf5, ndarrays etc.) for MCMC traces. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from utils import display_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Changes to the PyMC3 Code Base\n",
    "\n",
    "I've only converted the code required to recreate this basic example [\"A Motivating Example: Linear Regression\"](http://docs.pymc.io/notebooks/getting_started#A-Motivating-Example:-Linear-Regression). However, some principles that I used to accomplish this, can hopefully be extended to the rest of the code base. \n",
    "\n",
    "I have a local version of the PyMC3 code base here called 'pymc3-master'. It consists of these modules: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m__init__.py\u001b[m\u001b[m       \u001b[34mdistributions\u001b[m\u001b[m     \u001b[31mmemoize.py\u001b[m\u001b[m        \u001b[31mtheanof.py\u001b[m\u001b[m\r\n",
      "\u001b[34m__pycache__\u001b[m\u001b[m       \u001b[34mexamples\u001b[m\u001b[m          \u001b[31mmodel.py\u001b[m\u001b[m          \u001b[34mtuning\u001b[m\u001b[m\r\n",
      "\u001b[34mbackends\u001b[m\u001b[m          \u001b[31mexceptions.py\u001b[m\u001b[m     \u001b[34mplots\u001b[m\u001b[m             \u001b[31mutil.py\u001b[m\u001b[m\r\n",
      "\u001b[34mbackends_symbolic\u001b[m\u001b[m \u001b[34mexternal\u001b[m\u001b[m          \u001b[31msampling.py\u001b[m\u001b[m       \u001b[34mvariational\u001b[m\u001b[m\r\n",
      "\u001b[31mblocking.py\u001b[m\u001b[m       \u001b[34mglm\u001b[m\u001b[m               \u001b[31mstats.py\u001b[m\u001b[m          \u001b[31mvartypes.py\u001b[m\u001b[m\r\n",
      "\u001b[31mdata.py\u001b[m\u001b[m           \u001b[34mgp\u001b[m\u001b[m                \u001b[34mstep_methods\u001b[m\u001b[m\r\n",
      "\u001b[31mdiagnostics.py\u001b[m\u001b[m    \u001b[31mmath.py\u001b[m\u001b[m           \u001b[34mtests\u001b[m\u001b[m\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./pymc3-master/pymc3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run the basic example, I needed to alter: \n",
    "- `__init__.py`\n",
    "- `model.py`\n",
    "- `distributions/distribution.py`\n",
    "- `distributions/continuous.py`\n",
    "- `step_methods/arraystep.py`\n",
    "- `step_methods/metropolis.py`\n",
    "\n",
    "I'll go through the changes needed for each module in turn. \n",
    "\n",
    "I also heavily commented code that I did not want to change, but was imported with PyMC3's initial import. For example, I only kept the `Normal` distribution in `continuous.py`, and commented all the rest out. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A new module: 'Backends_Symbolic'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Much of the Theano functionality is either directly imported from Theano at the top of each module, or imported from\n",
    "PyMC3 modules such as `theanof.py` or `math.py`. I commented these imports and instead import all the necessary functionality from my new module called `backends_symbolic`. Here is an example, of how I comment out the imports and replace with a single import to my new backend. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">scipy.sparse</span> <span class=\"kn\">as</span> <span class=\"nn\">sps</span>\n",
       "\n",
       "<span class=\"c1\">#import theano.sparse as sparse</span>\n",
       "<span class=\"c1\">#from theano import theano, tensor as tt</span>\n",
       "<span class=\"c1\">#from theano.tensor.var import TensorVariable</span>\n",
       "<span class=\"c1\">#from pymc3.theanof import set_theano_conf, floatX</span>\n",
       "<span class=\"c1\">#from .theanof import gradient, hessian, inputvars, generator</span>\n",
       "<span class=\"c1\">#from pymc3.math import flatten_list</span>\n",
       "<span class=\"kn\">from</span> <span class=\"nn\">.</span> <span class=\"kn\">import</span> <span class=\"n\">backends_symbolic</span> <span class=\"k\">as</span> <span class=\"n\">S</span>\n",
       "\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">pymc3</span> <span class=\"kn\">as</span> <span class=\"nn\">pm</span>\n",
       "<span class=\"kn\">from</span> <span class=\"nn\">.memoize</span> <span class=\"kn\">import</span> <span class=\"n\">memoize</span><span class=\"p\">,</span> <span class=\"n\">WithMemoization</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_code('model.py',[8,20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within `symbolic_backends`, I have implementations of the same functions in two different modules, `backends_tf.py` and `backends_theano.py`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__init__.py        backends_tf.py     common.py\r\n",
      "\u001b[34m__pycache__\u001b[m\u001b[m        backends_theano.py\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./pymc3-master/pymc3/*backends_symbolic*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default symbolic backend is Theano, but TensorFlow can be specified by setting an environment variable `os.environ[\"PYMC_SYMB_BACKEND\"] = \"tensorflow\" `. This environment variable is referenced during the import of the module, so that either the TensorFlow or Theano implementations of each function are referenced. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"n\">_BACKEND</span> <span class=\"o\">=</span> <span class=\"s1\">&#39;theano&#39;</span>\n",
       "\n",
       "<span class=\"c1\"># Check environment for backend</span>\n",
       "<span class=\"k\">if</span> <span class=\"s1\">&#39;PYMC_SYMB_BACKEND&#39;</span> <span class=\"ow\">in</span> <span class=\"n\">os</span><span class=\"o\">.</span><span class=\"n\">environ</span><span class=\"p\">:</span>\n",
       "    <span class=\"n\">_backend</span> <span class=\"o\">=</span> <span class=\"n\">os</span><span class=\"o\">.</span><span class=\"n\">environ</span><span class=\"p\">[</span><span class=\"s1\">&#39;PYMC_SYMB_BACKEND&#39;</span><span class=\"p\">]</span>\n",
       "    <span class=\"n\">_BACKEND</span> <span class=\"o\">=</span> <span class=\"n\">_backend</span>\n",
       "\n",
       "<span class=\"k\">if</span> <span class=\"n\">_BACKEND</span> <span class=\"o\">==</span> <span class=\"s1\">&#39;theano&#39;</span><span class=\"p\">:</span>\n",
       "    <span class=\"n\">sys</span><span class=\"o\">.</span><span class=\"n\">stderr</span><span class=\"o\">.</span><span class=\"n\">write</span><span class=\"p\">(</span><span class=\"s1\">&#39;Using Theano backend.</span><span class=\"se\">\\n</span><span class=\"s1\">&#39;</span><span class=\"p\">)</span>\n",
       "    <span class=\"kn\">from</span> <span class=\"nn\">.backends_theano</span> <span class=\"kn\">import</span> <span class=\"o\">*</span>\n",
       "\n",
       "<span class=\"k\">elif</span> <span class=\"n\">_BACKEND</span> <span class=\"o\">==</span> <span class=\"s1\">&#39;tensorflow&#39;</span><span class=\"p\">:</span>\n",
       "    <span class=\"n\">sys</span><span class=\"o\">.</span><span class=\"n\">stderr</span><span class=\"o\">.</span><span class=\"n\">write</span><span class=\"p\">(</span><span class=\"s1\">&#39;Using TensorFlow backend.</span><span class=\"se\">\\n</span><span class=\"s1\">&#39;</span><span class=\"p\">)</span>\n",
       "    <span class=\"kn\">from</span> <span class=\"nn\">.backends_tf</span> <span class=\"kn\">import</span> <span class=\"o\">*</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/__init__.py'\n",
    "lines=[4,20]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changes needed for model.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The essential object of PyMC3 is a model, which consists of linked random variables. The two main random variables are ObservedRV and FreeRV, which have and do not have observed data, respectively. \n",
    "\n",
    "\n",
    "#### TensorVariables\n",
    "\n",
    "These random variable classes were originally a subclass of the `TensorVariable` in Theano, but now they differ depending on backend. In Theano, it's still a `TensorVariable`, and in TensorFlow, it's a `Variable`.  \n",
    "\n",
    "Here is an example of the class change in `models.py`. Note that I left a `##CHANGED:` comment everywhere that I adjusted the original PyMC3 code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"c1\">## CHANGED: Class now differs between theano and tensorflow</span>\n",
       "<span class=\"k\">class</span> <span class=\"nc\">FreeRV</span><span class=\"p\">(</span><span class=\"n\">Factor</span><span class=\"p\">,</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">TensorVariable</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Unobserved random variable that a model is specified in terms of.&quot;&quot;&quot;</span>\n",
       "\n",
       "    <span class=\"k\">def</span> <span class=\"fm\">__init__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"nb\">type</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">owner</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">index</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">name</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'model.py'\n",
    "lines=[1234,1239]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is the basically empty wrapper in the backends modules. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">tensorflow</span> <span class=\"kn\">as</span> <span class=\"nn\">tf</span>\n",
       "<span class=\"kn\">from</span> <span class=\"nn\">tensorflow</span> <span class=\"kn\">import</span> <span class=\"n\">Variable</span><span class=\"p\">,</span> <span class=\"n\">Tensor</span>\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">os</span>\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">sys</span>\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">inspect</span>\n",
       "\n",
       "<span class=\"kn\">from</span> <span class=\"nn\">.common</span> <span class=\"kn\">import</span> <span class=\"n\">floatx</span><span class=\"p\">,</span> <span class=\"n\">has_arg</span><span class=\"p\">,</span> <span class=\"n\">makeiter</span>\n",
       "\n",
       "<span class=\"n\">_SESSION</span> <span class=\"o\">=</span> <span class=\"bp\">None</span>\n",
       "\n",
       "<span class=\"c1\">### Variables and Variable Manipulation ###</span>\n",
       "\n",
       "<span class=\"k\">class</span> <span class=\"nc\">TensorVariable</span><span class=\"p\">(</span><span class=\"n\">Variable</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Main Class for FreeRV and ObservedRV</span>\n",
       "<span class=\"sd\">       It&#39;s a Variable in TensorFlow</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[1,17]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One implication of this change is that using a TensorFlow backend requires an `initial_value` be specified for all variables, even free random variables. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TensorVariable-related functions\n",
    "\n",
    "There are a few other tensor variable related functions that I needed to implement for both backends. I've added doc strings, but these functions are pretty self-explanatory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">TensorVariableType</span><span class=\"p\">(</span><span class=\"n\">dtype</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span><span class=\"n\">shape</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">name</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Returns the the type for the TensorVariable</span>\n",
       "<span class=\"sd\">       Formerly called TensorType</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"c1\">### TODO: may need to add broadcastable to match theano backend.</span>\n",
       "    <span class=\"k\">if</span> <span class=\"n\">dtype</span> <span class=\"ow\">is</span> <span class=\"bp\">None</span><span class=\"p\">:</span>\n",
       "        <span class=\"n\">dtype</span> <span class=\"o\">=</span> <span class=\"n\">floatx</span><span class=\"p\">()</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">Variable</span> <span class=\"c1\"># returns a class?</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">as_tensor_variable</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">,</span><span class=\"n\">name</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span><span class=\"n\">dtype</span><span class=\"o\">=</span><span class=\"n\">floatx</span><span class=\"p\">()):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Converts data, like numpy array, to a TensorVariable of appropriate type.</span>\n",
       "<span class=\"sd\">       Used for distribution parameters, like mu.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">convert_to_tensor</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">,</span> <span class=\"n\">dtype</span><span class=\"o\">=</span><span class=\"n\">dtype</span><span class=\"p\">,</span><span class=\"n\">name</span><span class=\"o\">=</span><span class=\"n\">name</span><span class=\"p\">))</span>\n",
       "\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">is_shared</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">):</span>\n",
       "    <span class=\"c1\">### tf Variables act like shared variables</span>\n",
       "    <span class=\"k\">return</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">Variable</span><span class=\"p\">)</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">is_variable</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">):</span>\n",
       "    <span class=\"k\">return</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">Variable</span><span class=\"p\">)</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">is_constant</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">):</span>\n",
       "    <span class=\"k\">return</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">constant</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[25,50]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changes needed for distribution.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Free and observed random variables in PyMC each come with `Distribution` classes. I only altered the `Normal` distribution in the `Continuous` distribution class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting Values from Symbolic Variables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distributions have functions, such as sample, that require getting the actual values from symbolic variables. \n",
    "\n",
    "Getting values differs in Theano and TensorFlow. In Theano, if a variable has a value, it can be obtained directly. The type of variable needs to be checked beforehand, however, because the call is slightly different. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">get_val</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Gets the current value of a symbolic variable</span>\n",
       "<span class=\"sd\">       Theano has different methods for getting the value depending on variable type</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">if</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tt</span><span class=\"o\">.</span><span class=\"n\">TensorVariable</span><span class=\"p\">):</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">var</span><span class=\"o\">.</span><span class=\"n\">tag</span><span class=\"o\">.</span><span class=\"n\">test_value</span>\n",
       "    <span class=\"k\">if</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tt</span><span class=\"o\">.</span><span class=\"n\">TensorConstant</span><span class=\"p\">):</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">var</span><span class=\"o\">.</span><span class=\"n\">value</span>\n",
       "    <span class=\"k\">if</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">tt</span><span class=\"o\">.</span><span class=\"n\">SharedVariable</span><span class=\"p\">):</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">var</span><span class=\"o\">.</span><span class=\"n\">get_value</span><span class=\"p\">()</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_theano.py'\n",
    "lines=[45,56]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorFlow, a variable needs to be fed to a session and the session needs to be run in order to return a value. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">get_val</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Gets the current value of a symbolic variable</span>\n",
       "<span class=\"sd\">       TensorFlow requires current session in order to get value.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">var</span><span class=\"o\">.</span><span class=\"n\">eval</span><span class=\"p\">(</span><span class=\"n\">session</span><span class=\"o\">=</span><span class=\"n\">get_session</span><span class=\"p\">()))</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[54,60]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorFlow, a variable needs to be fed to a session and the session needs to be run in order to return a value. \n",
    "\n",
    "#### TensorFlow Sessions\n",
    "\n",
    "The TensorFlow backend deals with instantiating a session the first time around and retrieving it when needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">get_session</span><span class=\"p\">():</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Returns the TF session to be used by the backend. Required to evaluate</span>\n",
       "<span class=\"sd\">    computations.</span>\n",
       "\n",
       "<span class=\"sd\">    Returns</span>\n",
       "<span class=\"sd\">    -------</span>\n",
       "<span class=\"sd\">        A TensorFlow session.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[80,88]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Functions\n",
    "\n",
    "Another major change was to replace the calls to `theano.function`, with a backend-agnostic implementation. Keras' implementation helped a lot here. \n",
    "\n",
    "For example, `distribution.py` has a function that draws values from a distribution. It uses `_compile_theano_function`, a memoized and wrapped version of `theano.function`, which returns a compiled Theano function.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>            <span class=\"c1\">## CHANGED: call the function class defined in the backend.</span>\n",
       "            <span class=\"c1\">#func = _compile_theano_function(param, variables)</span>\n",
       "            <span class=\"c1\">#return func(*values)</span>\n",
       "            <span class=\"n\">func</span> <span class=\"o\">=</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">function</span><span class=\"p\">(</span><span class=\"n\">param</span><span class=\"p\">,</span><span class=\"n\">variables</span><span class=\"p\">)</span>\n",
       "            <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">func</span><span class=\"p\">(</span><span class=\"o\">*</span><span class=\"n\">values</span><span class=\"p\">))</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'distributions/distribution.py'\n",
    "lines=[398,405]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I replaced this with a call to my backend. Like keras, I've wrapped `theano.function` into a class called `Function`. The initialization of the class upon instantiation compiles the function, and the call evaluates it. Here's the implementation in my backend.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">class</span> <span class=\"nc\">Function</span><span class=\"p\">(</span><span class=\"nb\">object</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;</span>\n",
       "<span class=\"sd\">       Runs a computation graph.</span>\n",
       "\n",
       "<span class=\"sd\">       This is a class to make syntax similar between theano.functions</span>\n",
       "<span class=\"sd\">       and tensorflow session.run().</span>\n",
       "\n",
       "<span class=\"sd\">       The initialization compiles the theano function. Calling the Function</span>\n",
       "<span class=\"sd\">       class calls the compiled theano function.</span>\n",
       "\n",
       "<span class=\"sd\">       See Keras.backends for similar implementation.</span>\n",
       "\n",
       "<span class=\"sd\">       Note: @memoize is now gone.</span>\n",
       "\n",
       "<span class=\"sd\">       Inputs:</span>\n",
       "<span class=\"sd\">       -------</span>\n",
       "<span class=\"sd\">       inputs: Feed placeholders to the computation graph.</span>\n",
       "<span class=\"sd\">       outputs: Output tensors to fetch.</span>\n",
       "<span class=\"sd\">       updates: Additional update ops to be run at function call.</span>\n",
       "<span class=\"sd\">       name: a name to help users identify what this function does.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">def</span> <span class=\"fm\">__init__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"n\">updates</span><span class=\"o\">=</span><span class=\"p\">[],</span> <span class=\"n\">name</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"o\">**</span><span class=\"n\">kwargs</span><span class=\"p\">):</span>\n",
       "        <span class=\"n\">unique_variables_to_update</span> <span class=\"o\">=</span> <span class=\"p\">{}</span>\n",
       "        <span class=\"k\">for</span> <span class=\"n\">v</span><span class=\"p\">,</span> <span class=\"n\">nv</span> <span class=\"ow\">in</span> <span class=\"n\">updates</span><span class=\"p\">:</span>\n",
       "            <span class=\"k\">if</span> <span class=\"n\">v</span> <span class=\"ow\">not</span> <span class=\"ow\">in</span> <span class=\"n\">unique_variables_to_update</span><span class=\"p\">:</span>\n",
       "                <span class=\"n\">unique_variables_to_update</span><span class=\"p\">[</span><span class=\"n\">v</span><span class=\"p\">]</span> <span class=\"o\">=</span> <span class=\"n\">nv</span>\n",
       "        <span class=\"n\">updates</span> <span class=\"o\">=</span> <span class=\"n\">unique_variables_to_update</span><span class=\"o\">.</span><span class=\"n\">items</span><span class=\"p\">()</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">function</span> <span class=\"o\">=</span> <span class=\"n\">theano</span><span class=\"o\">.</span><span class=\"n\">function</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"n\">updates</span><span class=\"o\">=</span><span class=\"n\">updates</span><span class=\"p\">,</span>\n",
       "                                        <span class=\"n\">allow_input_downcast</span><span class=\"o\">=</span><span class=\"bp\">True</span><span class=\"p\">,</span>\n",
       "                                        <span class=\"n\">on_unused_input</span><span class=\"o\">=</span><span class=\"s1\">&#39;ignore&#39;</span><span class=\"p\">,</span>\n",
       "                                        <span class=\"n\">name</span><span class=\"o\">=</span><span class=\"n\">name</span><span class=\"p\">,</span>\n",
       "                                        <span class=\"o\">**</span><span class=\"n\">kwargs</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">name</span> <span class=\"o\">=</span> <span class=\"n\">name</span>\n",
       "\n",
       "    <span class=\"k\">def</span> <span class=\"fm\">__call__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">inputs</span><span class=\"p\">):</span>\n",
       "        <span class=\"k\">return</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">function</span><span class=\"p\">(</span><span class=\"o\">**</span><span class=\"n\">inputs</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_theano.py'\n",
    "lines=[97,135]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like Keras, there is also an accompanying `function` function which instantiates the `Function` class. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">function</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"n\">updates</span><span class=\"o\">=</span><span class=\"p\">[],</span> <span class=\"o\">**</span><span class=\"n\">kwargs</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Instantiates a Function class. Also checks validity of arguments. &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">if</span> <span class=\"nb\">len</span><span class=\"p\">(</span><span class=\"n\">kwargs</span><span class=\"p\">)</span> <span class=\"o\">&gt;</span> <span class=\"mi\">0</span><span class=\"p\">:</span>\n",
       "        <span class=\"k\">for</span> <span class=\"n\">key</span> <span class=\"ow\">in</span> <span class=\"n\">kwargs</span><span class=\"o\">.</span><span class=\"n\">keys</span><span class=\"p\">():</span>\n",
       "            <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"n\">has_arg</span><span class=\"p\">(</span><span class=\"n\">theano</span><span class=\"o\">.</span><span class=\"n\">function</span><span class=\"p\">,</span> <span class=\"n\">key</span><span class=\"p\">,</span> <span class=\"bp\">True</span><span class=\"p\">):</span>\n",
       "                <span class=\"n\">msg</span> <span class=\"o\">=</span> <span class=\"s1\">&#39;Invalid argument &quot;</span><span class=\"si\">%s</span><span class=\"s1\">&quot; passed to K.function with Theano backend&#39;</span> <span class=\"o\">%</span> <span class=\"n\">key</span>\n",
       "                <span class=\"k\">raise</span> <span class=\"ne\">ValueError</span><span class=\"p\">(</span><span class=\"n\">msg</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">Function</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"n\">updates</span><span class=\"o\">=</span><span class=\"n\">updates</span><span class=\"p\">,</span> <span class=\"o\">**</span><span class=\"n\">kwargs</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_theano.py'\n",
    "lines=[135,145]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This implementation allows us to use the same syntax for TensorFlow, which doesn't compile a function per se. Here, the intialization of `Function` relates the inputs, outputs, updates, and session variable. The call then evaluates the output variables in the session. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">class</span> <span class=\"nc\">Function</span><span class=\"p\">(</span><span class=\"nb\">object</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;</span>\n",
       "<span class=\"sd\">       Runs a computation graph.</span>\n",
       "\n",
       "<span class=\"sd\">       This is a class to make syntax similar between theano.functions</span>\n",
       "<span class=\"sd\">       and tensorflow session.run().</span>\n",
       "\n",
       "<span class=\"sd\">       The initialization associates all the inputs, outputs, and update operations.</span>\n",
       "<span class=\"sd\">       Calling the Function evaluates the comptutation.</span>\n",
       "\n",
       "<span class=\"sd\">       See Keras.backends for similar implementation.</span>\n",
       "\n",
       "<span class=\"sd\">       Note: @memoize is now gone.</span>\n",
       "\n",
       "<span class=\"sd\">       Inputs:</span>\n",
       "<span class=\"sd\">       -------</span>\n",
       "<span class=\"sd\">       inputs: Feed placeholders to the computation graph.</span>\n",
       "<span class=\"sd\">       outputs: Output tensors to fetch.</span>\n",
       "<span class=\"sd\">       updates: Additional update ops to be run at function call.</span>\n",
       "<span class=\"sd\">       name: a name to help users identify what this function does.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "\n",
       "    <span class=\"k\">def</span> <span class=\"fm\">__init__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"n\">updates</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">name</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"o\">**</span><span class=\"n\">session_kwargs</span><span class=\"p\">):</span>\n",
       "        <span class=\"n\">updates</span> <span class=\"o\">=</span> <span class=\"n\">updates</span> <span class=\"ow\">or</span> <span class=\"p\">[]</span>\n",
       "        <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"p\">(</span><span class=\"nb\">list</span><span class=\"p\">,</span> <span class=\"nb\">tuple</span><span class=\"p\">)):</span>\n",
       "            <span class=\"k\">raise</span> <span class=\"ne\">TypeError</span><span class=\"p\">(</span><span class=\"s1\">&#39;`inputs` to a TensorFlow backend function &#39;</span>\n",
       "                            <span class=\"s1\">&#39;should be a list or tuple.&#39;</span><span class=\"p\">)</span>\n",
       "        <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">outputs</span><span class=\"p\">,</span> <span class=\"p\">(</span><span class=\"nb\">list</span><span class=\"p\">,</span> <span class=\"nb\">tuple</span><span class=\"p\">)):</span>\n",
       "            <span class=\"k\">raise</span> <span class=\"ne\">TypeError</span><span class=\"p\">(</span><span class=\"s1\">&#39;`outputs` of a TensorFlow backend function &#39;</span>\n",
       "                            <span class=\"s1\">&#39;should be a list or tuple.&#39;</span><span class=\"p\">)</span>\n",
       "        <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">updates</span><span class=\"p\">,</span> <span class=\"p\">(</span><span class=\"nb\">list</span><span class=\"p\">,</span> <span class=\"nb\">tuple</span><span class=\"p\">)):</span>\n",
       "            <span class=\"k\">raise</span> <span class=\"ne\">TypeError</span><span class=\"p\">(</span><span class=\"s1\">&#39;`updates` in a TensorFlow backend function &#39;</span>\n",
       "                            <span class=\"s1\">&#39;should be a list or tuple.&#39;</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">inputs</span> <span class=\"o\">=</span> <span class=\"nb\">list</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">outputs</span> <span class=\"o\">=</span> <span class=\"nb\">list</span><span class=\"p\">(</span><span class=\"n\">outputs</span><span class=\"p\">)</span>\n",
       "        <span class=\"k\">with</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">control_dependencies</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">outputs</span><span class=\"p\">):</span>\n",
       "            <span class=\"n\">updates_ops</span> <span class=\"o\">=</span> <span class=\"p\">[]</span>\n",
       "            <span class=\"k\">for</span> <span class=\"n\">update</span> <span class=\"ow\">in</span> <span class=\"n\">updates</span><span class=\"p\">:</span>\n",
       "                <span class=\"k\">if</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">update</span><span class=\"p\">,</span> <span class=\"nb\">tuple</span><span class=\"p\">):</span>\n",
       "                    <span class=\"n\">p</span><span class=\"p\">,</span> <span class=\"n\">new_p</span> <span class=\"o\">=</span> <span class=\"n\">update</span>\n",
       "                    <span class=\"n\">updates_ops</span><span class=\"o\">.</span><span class=\"n\">append</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">assign</span><span class=\"p\">(</span><span class=\"n\">p</span><span class=\"p\">,</span> <span class=\"n\">new_p</span><span class=\"p\">))</span>\n",
       "                <span class=\"k\">else</span><span class=\"p\">:</span>\n",
       "                    <span class=\"c1\"># assumed already an op</span>\n",
       "                    <span class=\"n\">updates_ops</span><span class=\"o\">.</span><span class=\"n\">append</span><span class=\"p\">(</span><span class=\"n\">update</span><span class=\"p\">)</span>\n",
       "            <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">updates_op</span> <span class=\"o\">=</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">group</span><span class=\"p\">(</span><span class=\"o\">*</span><span class=\"n\">updates_ops</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">name</span> <span class=\"o\">=</span> <span class=\"n\">name</span>\n",
       "        <span class=\"c1\"># additional tensor substitutions</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">feed_dict</span> <span class=\"o\">=</span> <span class=\"n\">session_kwargs</span><span class=\"o\">.</span><span class=\"n\">pop</span><span class=\"p\">(</span><span class=\"s1\">&#39;feed_dict&#39;</span><span class=\"p\">,</span> <span class=\"p\">{})</span>\n",
       "        <span class=\"c1\"># additional operations</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">fetches</span> <span class=\"o\">=</span> <span class=\"n\">session_kwargs</span><span class=\"o\">.</span><span class=\"n\">pop</span><span class=\"p\">(</span><span class=\"s1\">&#39;fetches&#39;</span><span class=\"p\">,</span> <span class=\"p\">[])</span>\n",
       "        <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">fetches</span><span class=\"p\">,</span> <span class=\"nb\">list</span><span class=\"p\">):</span>\n",
       "            <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">fetches</span> <span class=\"o\">=</span> <span class=\"p\">[</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">fetches</span><span class=\"p\">]</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">session_kwargs</span> <span class=\"o\">=</span> <span class=\"n\">session_kwargs</span>\n",
       "\n",
       "    <span class=\"k\">def</span> <span class=\"fm\">__call__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">inputs</span><span class=\"p\">):</span>\n",
       "        <span class=\"k\">if</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"p\">(</span><span class=\"nb\">list</span><span class=\"p\">,</span> <span class=\"nb\">tuple</span><span class=\"p\">)):</span>\n",
       "            <span class=\"n\">feed_dict</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">feed_dict</span><span class=\"o\">.</span><span class=\"n\">copy</span><span class=\"p\">()</span>\n",
       "            <span class=\"k\">for</span> <span class=\"n\">tensor</span><span class=\"p\">,</span> <span class=\"n\">value</span> <span class=\"ow\">in</span> <span class=\"nb\">zip</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">inputs</span><span class=\"p\">,</span> <span class=\"n\">inputs</span><span class=\"p\">):</span>\n",
       "                <span class=\"n\">feed_dict</span><span class=\"p\">[</span><span class=\"n\">tensor</span><span class=\"p\">]</span> <span class=\"o\">=</span> <span class=\"n\">value</span>\n",
       "        <span class=\"k\">elif</span> <span class=\"nb\">isinstance</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,(</span><span class=\"nb\">dict</span><span class=\"p\">)):</span>\n",
       "            <span class=\"n\">feed_dict</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">feed_dict</span><span class=\"o\">.</span><span class=\"n\">copy</span><span class=\"p\">()</span>\n",
       "            <span class=\"c1\"># This maps inputs to those in self.inputs.</span>\n",
       "            <span class=\"c1\"># The inputs come in in the order of the variables, but self.inputs</span>\n",
       "            <span class=\"c1\"># is ordered based on which variable is main input (ie. which distribution)</span>\n",
       "            <span class=\"k\">for</span> <span class=\"n\">tensor</span> <span class=\"ow\">in</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">inputs</span><span class=\"p\">:</span>\n",
       "                <span class=\"n\">feed_dict</span><span class=\"p\">[</span><span class=\"n\">tensor</span><span class=\"p\">]</span> <span class=\"o\">=</span> <span class=\"n\">inputs</span><span class=\"p\">[</span><span class=\"nb\">str</span><span class=\"p\">(</span><span class=\"n\">tensor</span><span class=\"p\">)]</span>\n",
       "        <span class=\"k\">else</span><span class=\"p\">:</span>\n",
       "            <span class=\"k\">print</span><span class=\"p\">(</span><span class=\"s1\">&#39;error: input needs to be dict,list,or tuple&#39;</span><span class=\"p\">)</span>\n",
       "\n",
       "        <span class=\"n\">fetches</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">outputs</span> <span class=\"o\">+</span> <span class=\"p\">[</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">updates_op</span><span class=\"p\">]</span> <span class=\"o\">+</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">fetches</span>\n",
       "        <span class=\"n\">session</span> <span class=\"o\">=</span> <span class=\"n\">get_session</span><span class=\"p\">()</span>\n",
       "        <span class=\"n\">updated</span> <span class=\"o\">=</span> <span class=\"n\">session</span><span class=\"o\">.</span><span class=\"n\">run</span><span class=\"p\">(</span><span class=\"n\">fetches</span><span class=\"o\">=</span><span class=\"n\">fetches</span><span class=\"p\">,</span> <span class=\"n\">feed_dict</span><span class=\"o\">=</span><span class=\"n\">feed_dict</span><span class=\"p\">)</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">updated</span><span class=\"p\">[:</span><span class=\"nb\">len</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">outputs</span><span class=\"p\">)]</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[130,205]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `Function` class is used for when PyMC's `Model` class compiles the likelihood function using the `logp` output variables and the various input variables.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changes needed for continuous.py\n",
    "\n",
    "The `Normal` distribution class contains the parameters of the distribution. Here, for example, there is the standard deviation and mean for the normal distribution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>    <span class=\"k\">def</span> <span class=\"fm\">__init__</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">mu</span><span class=\"o\">=</span><span class=\"mf\">0.0</span><span class=\"p\">,</span> <span class=\"n\">sd</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">tau</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"o\">**</span><span class=\"n\">kwargs</span><span class=\"p\">):</span>\n",
       "        <span class=\"n\">tau</span><span class=\"p\">,</span> <span class=\"n\">sd</span> <span class=\"o\">=</span> <span class=\"n\">get_tau_sd</span><span class=\"p\">(</span><span class=\"n\">tau</span><span class=\"o\">=</span><span class=\"n\">tau</span><span class=\"p\">,</span> <span class=\"n\">sd</span><span class=\"o\">=</span><span class=\"n\">sd</span><span class=\"p\">)</span>\n",
       "\n",
       "        <span class=\"c1\">## CHANGED:</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">sd</span> <span class=\"o\">=</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">as_tensor_variable</span><span class=\"p\">(</span><span class=\"n\">sd</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">tau</span> <span class=\"o\">=</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">as_tensor_variable</span><span class=\"p\">(</span><span class=\"n\">tau</span><span class=\"p\">)</span>\n",
       "        <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">mean</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">median</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">mode</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">mu</span> <span class=\"o\">=</span> <span class=\"n\">mu</span> <span class=\"o\">=</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">as_tensor_variable</span><span class=\"p\">(</span><span class=\"n\">mu</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'distributions/continuous.py'\n",
    "lines=[297,304]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also contains the symbolic definition of the loglikelihood. These are the 'outputs' in the computation graph and with the 'inputs' will be be compiled into a Theano function or fed to a TensorFlow session. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>    <span class=\"k\">def</span> <span class=\"nf\">logp</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">value</span><span class=\"p\">):</span>\n",
       "        <span class=\"n\">sd</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">sd</span>\n",
       "        <span class=\"n\">tau</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">tau</span>\n",
       "        <span class=\"n\">mu</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">mu</span>\n",
       "        <span class=\"c1\">## CHANGED: Unfortunately had to move stuff like bound into the backend and out of bound.py</span>\n",
       "        <span class=\"c1\"># return bound((-tau * (value - mu)**2 + tt.log(tau / np.pi / 2.)) / 2.,</span>\n",
       "        <span class=\"c1\">#              sd &gt; 0)</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">bound</span><span class=\"p\">((</span><span class=\"o\">-</span><span class=\"n\">tau</span> <span class=\"o\">*</span> <span class=\"p\">(</span><span class=\"n\">value</span> <span class=\"o\">-</span> <span class=\"n\">mu</span><span class=\"p\">)</span><span class=\"o\">**</span><span class=\"mi\">2</span> <span class=\"o\">+</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">log</span><span class=\"p\">(</span><span class=\"n\">tau</span> <span class=\"o\">/</span> <span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">pi</span> <span class=\"o\">/</span> <span class=\"mf\">2.</span><span class=\"p\">))</span> <span class=\"o\">/</span> <span class=\"mf\">2.</span><span class=\"p\">,</span>\n",
       "                     <span class=\"n\">sd</span> <span class=\"o\">&gt;</span> <span class=\"mi\">0</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'distributions/continuous.py'\n",
    "lines=[322,331]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Math\n",
    "All of the basic mathematical functions required for specifying the likelihood have been moved to the backend. Here are some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">tsum</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">,</span><span class=\"n\">axis</span><span class=\"o\">=</span><span class=\"bp\">None</span><span class=\"p\">,</span> <span class=\"n\">keepdims</span><span class=\"o\">=</span><span class=\"bp\">False</span><span class=\"p\">):</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">reduce_sum</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">,</span><span class=\"n\">axis</span><span class=\"p\">,</span><span class=\"n\">keepdims</span><span class=\"p\">))</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">sqrt</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">):</span>\n",
       "    <span class=\"n\">zero</span> <span class=\"o\">=</span> <span class=\"n\">_to_tensor</span><span class=\"p\">(</span><span class=\"mf\">0.</span><span class=\"p\">,</span> <span class=\"n\">x</span><span class=\"o\">.</span><span class=\"n\">dtype</span><span class=\"o\">.</span><span class=\"n\">base_dtype</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">inf</span> <span class=\"o\">=</span> <span class=\"n\">_to_tensor</span><span class=\"p\">(</span><span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">inf</span><span class=\"p\">,</span> <span class=\"n\">x</span><span class=\"o\">.</span><span class=\"n\">dtype</span><span class=\"o\">.</span><span class=\"n\">base_dtype</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">x</span> <span class=\"o\">=</span> <span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">clip_by_value</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">,</span> <span class=\"n\">zero</span><span class=\"p\">,</span> <span class=\"n\">inf</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">sqrt</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">))</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">log</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">):</span>\n",
       "   <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">log</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">))</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">grad</span><span class=\"p\">(</span><span class=\"n\">x</span><span class=\"p\">):</span>\n",
       "    <span class=\"k\">return</span><span class=\"p\">(</span><span class=\"n\">tf</span><span class=\"o\">.</span><span class=\"n\">gradients</span><span class=\"p\">(</span><span class=\"n\">loss</span><span class=\"p\">,</span> <span class=\"n\">variables</span><span class=\"p\">,</span> <span class=\"n\">colocate_gradients_with_ops</span><span class=\"o\">=</span><span class=\"bp\">True</span><span class=\"p\">))</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[220,234]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changes needed for arraystep.py\n",
    "\n",
    "The MCMC sampling in pymc is done by taking what's called a 'step' in parameter space from the current parameter values to a new set of values. The `ArrayStepShared` class takes one step at a time for each parameter. \n",
    "\n",
    "Each parameter has it's own instantiation of a particular step method, for example the Metropolis-Hasting method. Therefore the `step` function printed below from `ArrayStepShared` gets called for each variable in the model. \n",
    "\n",
    "For now, I've hacked together a way to implement the step in both Theano and TensorFlow. The simplest solution for now was to write a conditional for the different backends. In Theano, we only pass one input variable to the step method. In TensorFlow, I'm passing all the variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>    <span class=\"k\">def</span> <span class=\"nf\">step</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">point</span><span class=\"p\">):</span>\n",
       "\n",
       "        <span class=\"c1\">## CHANGED: At the moment, this is lazy.</span>\n",
       "        <span class=\"c1\">## But I implemented function for calculating change in logliklihood ...</span>\n",
       "        <span class=\"c1\">## for each step in each language differently.</span>\n",
       "        <span class=\"k\">if</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">backend</span><span class=\"p\">()</span><span class=\"o\">==</span><span class=\"s1\">&#39;theano&#39;</span><span class=\"p\">:</span>\n",
       "            <span class=\"k\">for</span> <span class=\"n\">var</span><span class=\"p\">,</span> <span class=\"n\">share</span> <span class=\"ow\">in</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">shared</span><span class=\"o\">.</span><span class=\"n\">items</span><span class=\"p\">():</span>\n",
       "                <span class=\"n\">share</span><span class=\"o\">.</span><span class=\"n\">set_value</span><span class=\"p\">(</span><span class=\"n\">point</span><span class=\"p\">[</span><span class=\"n\">var</span><span class=\"p\">])</span>\n",
       "\n",
       "            <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">bij</span> <span class=\"o\">=</span> <span class=\"n\">DictToArrayBijection</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">ordering</span><span class=\"p\">,</span> <span class=\"n\">point</span><span class=\"p\">)</span>\n",
       "\n",
       "            <span class=\"k\">if</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">generates_stats</span><span class=\"p\">:</span>\n",
       "                <span class=\"n\">apoint</span><span class=\"p\">,</span> <span class=\"n\">stats</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">astep</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">bij</span><span class=\"o\">.</span><span class=\"n\">map</span><span class=\"p\">(</span><span class=\"n\">point</span><span class=\"p\">))</span>\n",
       "                <span class=\"k\">return</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">bij</span><span class=\"o\">.</span><span class=\"n\">rmap</span><span class=\"p\">(</span><span class=\"n\">apoint</span><span class=\"p\">),</span> <span class=\"n\">stats</span>\n",
       "            <span class=\"k\">else</span><span class=\"p\">:</span>\n",
       "                <span class=\"n\">apoint</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">astep</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">bij</span><span class=\"o\">.</span><span class=\"n\">map</span><span class=\"p\">(</span><span class=\"n\">point</span><span class=\"p\">))</span> <span class=\"c1\"># it&#39;s only passing one variables value in becaues the others are shared.</span>\n",
       "                <span class=\"k\">return</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">bij</span><span class=\"o\">.</span><span class=\"n\">rmap</span><span class=\"p\">(</span><span class=\"n\">apoint</span><span class=\"p\">)</span>\n",
       "\n",
       "        <span class=\"k\">elif</span> <span class=\"n\">S</span><span class=\"o\">.</span><span class=\"n\">backend</span><span class=\"p\">()</span><span class=\"o\">==</span><span class=\"s1\">&#39;tensorflow&#39;</span><span class=\"p\">:</span>\n",
       "            <span class=\"n\">apoint</span><span class=\"p\">,</span> <span class=\"n\">stats</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">astep</span><span class=\"p\">(</span><span class=\"n\">point</span><span class=\"p\">)</span>\n",
       "            <span class=\"k\">return</span> <span class=\"n\">apoint</span><span class=\"p\">,</span> <span class=\"n\">stats</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'step_methods/arraystep.py'\n",
    "lines=[162,184]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changes needed for metropolis.py\n",
    "\n",
    "A Metropolis-Hasting step works as follows: \n",
    "- For a particular variable, and random step in parameter space is proposed from some proposal distribution (usually a normal distribution). \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>        <span class=\"n\">delta</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">proposal_dist</span><span class=\"p\">()</span> <span class=\"o\">*</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">scaling</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'step_methods/metropolis.py'\n",
    "lines=[163,164]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This proposed step `delta` is added to the original parameter value `q0`, to create `q`. \n",
    "- Then the change in the models log likelihood, `delta_logp`, is calculated.\n",
    "- If the change in likelihood is greater than some threshold, the new parameter value `q` is accepted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>                <span class=\"n\">q</span> <span class=\"o\">=</span> <span class=\"n\">SS</span><span class=\"o\">.</span><span class=\"n\">floatX</span><span class=\"p\">(</span><span class=\"n\">q0</span><span class=\"o\">+</span><span class=\"n\">delta</span><span class=\"p\">)</span>\n",
       "\n",
       "            <span class=\"n\">accept</span> <span class=\"o\">=</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">delta_logp</span><span class=\"p\">(</span><span class=\"n\">q</span><span class=\"p\">,</span> <span class=\"n\">q0</span><span class=\"p\">)</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'step_methods/metropolis.py'\n",
    "lines=[178,182]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've moved the calculation of the change in log likelihood to the symbolic backend. A compiled function gets associated with the metroplis step method upon its instantiation for each variable.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">delta_logp</span><span class=\"p\">(</span><span class=\"n\">logp</span><span class=\"p\">,</span> <span class=\"nb\">vars</span><span class=\"p\">,</span> <span class=\"n\">shared</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Returns a compiled theano function for change in logp</span>\n",
       "<span class=\"sd\">       The function will take 2 versions of the input that originally</span>\n",
       "<span class=\"sd\">       would go to logp.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "\n",
       "    <span class=\"p\">[</span><span class=\"n\">logp0</span><span class=\"p\">],</span> <span class=\"n\">inarray0</span> <span class=\"o\">=</span> <span class=\"n\">join_nonshared_inputs</span><span class=\"p\">([</span><span class=\"n\">logp</span><span class=\"p\">],</span> <span class=\"nb\">vars</span><span class=\"p\">,</span> <span class=\"n\">shared</span><span class=\"p\">)</span>\n",
       "\n",
       "    <span class=\"n\">tensor_type</span> <span class=\"o\">=</span> <span class=\"n\">inarray0</span><span class=\"o\">.</span><span class=\"n\">type</span>\n",
       "    <span class=\"n\">inarray1</span> <span class=\"o\">=</span> <span class=\"n\">tensor_type</span><span class=\"p\">(</span><span class=\"s1\">&#39;inarray1&#39;</span><span class=\"p\">)</span>\n",
       "\n",
       "    <span class=\"n\">logp1</span> <span class=\"o\">=</span> <span class=\"n\">CallableTensor</span><span class=\"p\">(</span><span class=\"n\">logp0</span><span class=\"p\">)(</span><span class=\"n\">inarray1</span><span class=\"p\">)</span>\n",
       "\n",
       "    <span class=\"n\">f</span> <span class=\"o\">=</span> <span class=\"n\">theano</span><span class=\"o\">.</span><span class=\"n\">function</span><span class=\"p\">([</span><span class=\"n\">inarray1</span><span class=\"p\">,</span> <span class=\"n\">inarray0</span><span class=\"p\">],</span> <span class=\"n\">logp1</span> <span class=\"o\">-</span> <span class=\"n\">logp0</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">f</span><span class=\"o\">.</span><span class=\"n\">trust_input</span> <span class=\"o\">=</span> <span class=\"bp\">True</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">f</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_theano.py'\n",
    "lines=[312,330]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For TensorFlow, I've implemented it slightly differently at the moment. Instead of having a `delta_logp` function, I just call the `logp` function twice, once for the old and new values, and calculate the difference. This is slower, but was an easier first pass. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span>        <span class=\"k\">elif</span> <span class=\"n\">SS</span><span class=\"o\">.</span><span class=\"n\">backend</span><span class=\"p\">()</span><span class=\"o\">==</span><span class=\"s1\">&#39;tensorflow&#39;</span><span class=\"p\">:</span>\n",
       "\n",
       "            <span class=\"n\">delta</span> <span class=\"o\">=</span> <span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">squeeze</span><span class=\"p\">(</span><span class=\"n\">delta</span><span class=\"p\">)</span>\n",
       "            <span class=\"n\">q</span> <span class=\"o\">=</span> <span class=\"n\">q0</span><span class=\"o\">.</span><span class=\"n\">copy</span><span class=\"p\">()</span>\n",
       "\n",
       "            <span class=\"k\">assert</span> <span class=\"nb\">len</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">vars</span><span class=\"p\">)</span><span class=\"o\">==</span><span class=\"mi\">1</span>\n",
       "\n",
       "            <span class=\"c1\">## This now takes in all variables in the model</span>\n",
       "            <span class=\"c1\">## and decides which to update based on the function object, which</span>\n",
       "            <span class=\"c1\">## had the order in it.</span>\n",
       "            <span class=\"n\">varname_to_update</span> <span class=\"o\">=</span> <span class=\"nb\">str</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">vars</span><span class=\"p\">[</span><span class=\"mi\">0</span><span class=\"p\">])</span> <span class=\"c1\"># decide which variable to update</span>\n",
       "            <span class=\"n\">q</span><span class=\"p\">[</span><span class=\"n\">varname_to_update</span><span class=\"p\">]</span><span class=\"o\">=</span><span class=\"n\">q0</span><span class=\"p\">[</span><span class=\"n\">varname_to_update</span><span class=\"p\">]</span><span class=\"o\">+</span><span class=\"n\">delta</span>\n",
       "\n",
       "            <span class=\"c1\"># q,q0 are dictionaries here with old and proposed values for the variable being updated</span>\n",
       "            <span class=\"c1\"># the other variables have the same value.</span>\n",
       "\n",
       "            <span class=\"c1\"># calculate logp for each point rather than the difference</span>\n",
       "            <span class=\"c1\"># (this was easier to do first in TensorFlow)</span>\n",
       "            <span class=\"n\">accept</span> <span class=\"o\">=</span> <span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">squeeze</span><span class=\"p\">(</span><span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">array</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">logp</span><span class=\"p\">(</span><span class=\"n\">q</span><span class=\"p\">))</span><span class=\"o\">-</span><span class=\"n\">np</span><span class=\"o\">.</span><span class=\"n\">array</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">logp</span><span class=\"p\">(</span><span class=\"n\">q0</span><span class=\"p\">)))</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'step_methods/metropolis.py'\n",
    "lines=[184,204]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, both of these functions rely on the `Function` class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
       ".highlight  { background: #f8f8f8; }\n",
       ".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
       ".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
       ".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".highlight .o { color: #666666 } /* Operator */\n",
       ".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       ".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       ".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       ".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       ".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       ".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       ".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".highlight .ge { font-style: italic } /* Generic.Emph */\n",
       ".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
       ".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
       ".highlight .go { color: #888888 } /* Generic.Output */\n",
       ".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
       ".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
       ".highlight .m { color: #666666 } /* Literal.Number */\n",
       ".highlight .s { color: #BA2121 } /* Literal.String */\n",
       ".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
       ".highlight .nb { color: #008000 } /* Name.Builtin */\n",
       ".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".highlight .no { color: #880000 } /* Name.Constant */\n",
       ".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       ".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       ".highlight .nf { color: #0000FF } /* Name.Function */\n",
       ".highlight .nl { color: #A0A000 } /* Name.Label */\n",
       ".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".highlight .nv { color: #19177C } /* Name.Variable */\n",
       ".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       ".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       ".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
       ".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       ".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">logp</span><span class=\"p\">(</span><span class=\"n\">logp</span><span class=\"p\">,</span> <span class=\"nb\">vars</span><span class=\"p\">,</span> <span class=\"n\">shared</span><span class=\"p\">):</span>\n",
       "    <span class=\"sd\">&quot;&quot;&quot;Returns a Function for logp with the inputs for the current stepself.</span>\n",
       "<span class=\"sd\">       Order matters. The variable being updated is first, and then the other</span>\n",
       "<span class=\"sd\">       variables in the model are in shared.</span>\n",
       "\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"n\">inputs</span> <span class=\"o\">=</span> <span class=\"nb\">vars</span><span class=\"o\">+</span><span class=\"p\">[</span><span class=\"n\">v</span> <span class=\"k\">for</span> <span class=\"n\">v</span> <span class=\"ow\">in</span> <span class=\"n\">shared</span><span class=\"o\">.</span><span class=\"n\">values</span><span class=\"p\">()]</span>\n",
       "    <span class=\"n\">outputs</span> <span class=\"o\">=</span> <span class=\"p\">[</span><span class=\"n\">logp</span><span class=\"p\">]</span>\n",
       "    <span class=\"n\">f</span> <span class=\"o\">=</span> <span class=\"n\">function</span><span class=\"p\">(</span><span class=\"n\">inputs</span><span class=\"p\">,</span><span class=\"n\">outputs</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">f</span>\n",
       "</pre></div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = 'backends_symbolic/backends_tf.py'\n",
    "lines=[283,294]\n",
    "display_code(file,lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Case: A Motivating Example: Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running with original pymc3 code (restart kernel)\n",
    "\n",
    "Here I import an unaltered PyMC3 package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "sys.path.append('/Users/chris/anaconda/envs/env_pymc3/lib/python3.6/')\n",
    "sys.path.append('/Users/chris/anaconda/envs/env_pymc3/lib/python3.6/site-packages/')\n",
    "import pymc3 as pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize random number generator\n",
    "np.random.seed(123)\n",
    "\n",
    "# True parameter values\n",
    "alpha, sigma = 1, 1; beta = [1, 2.5]\n",
    "\n",
    "# Predictor variable\n",
    "X1 = np.random.randn(100)\n",
    "X2 = np.random.randn(100) * 0.2\n",
    "\n",
    "# Simulate outcome variable\n",
    "Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(100)*sigma\n",
    "Y=Y.astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequential sampling (1 chains in 1 job)\n",
      "CompoundStep\n",
      ">Metropolis: [beta]\n",
      ">Metropolis: [alpha]\n",
      "100%|██████████| 5500/5500 [00:01<00:00, 3819.80it/s]\n",
      "Only one chain was sampled, this makes it impossible to run some convergence checks\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9054529\n",
      "[0.9458999 2.6006136]\n"
     ]
    }
   ],
   "source": [
    "basic_model2 = pm.Model()\n",
    "with basic_model2:\n",
    "    alpha = pm.Normal(name='alpha',mu=0.0, sd=10.0) \n",
    "    beta = pm.Normal(name='beta', mu=0.0, sd=10.0, shape=2)\n",
    "\n",
    "    mu = alpha + beta[0]*X1 + beta[1]*X2\n",
    "\n",
    "    Y_obs = pm.Normal(name='Y_obs',mu=mu, sd=1.0, observed=Y)\n",
    "\n",
    "    step = pm.Metropolis()\n",
    "    trace = pm.sample(5000,step=step,cores=1,chains=1) # switched off parallel for simplifications. \n",
    "\n",
    "print(np.mean(trace['alpha']))\n",
    "print(np.mean(trace['beta'],axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the estimated parameter values for alpha and beta. They are reasonably close to the true values of [1] and [1, 2.5]. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running with Theano backend (restart kernel for a fresh import)\n",
    "\n",
    "Here I import the local PyMC3 package that I changed. I set the backend to be Theano. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n",
      "/Users/chris/anaconda/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "os.environ[\"PYMC_SYMB_BACKEND\"] = \"theano\" \n",
    "sys.path.append('./pymc3-master/')\n",
    "import pymc3 as pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize random number generator\n",
    "np.random.seed(123)\n",
    "\n",
    "# True parameter values\n",
    "alpha, sigma = 1, 1; beta = [1, 2.5]\n",
    "\n",
    "# Predictor variable\n",
    "X1 = np.random.randn(100)\n",
    "X2 = np.random.randn(100) * 0.2\n",
    "\n",
    "# Simulate outcome variable\n",
    "Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(100)*sigma\n",
    "Y=Y.astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequential sampling (1 chains in 1 job)\n",
      "CompoundStep\n",
      ">Metropolis: [beta]\n",
      ">Metropolis: [alpha]\n",
      "100%|██████████| 5500/5500 [00:02<00:00, 2738.67it/s]\n",
      "Only one chain was sampled, this makes it impossible to run some convergence checks\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9054529\n",
      "[0.9458999 2.6006136]\n"
     ]
    }
   ],
   "source": [
    "basic_model2 = pm.Model()\n",
    "with basic_model2:\n",
    "    alpha = pm.Normal(name='alpha',mu=0.0, sd=10.0) \n",
    "    beta = pm.Normal(name='beta', mu=0.0, sd=10.0, shape=2)\n",
    "\n",
    "    mu = alpha + beta[0]*X1 + beta[1]*X2\n",
    "\n",
    "    Y_obs = pm.Normal(name='Y_obs',mu=mu, sd=1.0, observed=Y)\n",
    "\n",
    "    step = pm.Metropolis()\n",
    "    trace = pm.sample(5000,step=step,cores=1,chains=1) # switched off parallel for simplifications. \n",
    "\n",
    "print(np.mean(trace['alpha']))\n",
    "print(np.mean(trace['beta'],axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that that my code base produces the same answer as the original PyMC3 code base. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running with TensorFlow backend (restart kernel for a fresh import)\n",
    "\n",
    "After restarting the kernel, I import the local PyMC3 package again, but now I set the backend to be TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/Users/chris/anaconda/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "sys.path.append('./pymc3-master/')\n",
    "os.environ[\"PYMC_SYMB_BACKEND\"] = \"tensorflow\" \n",
    "import pymc3 as pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize random number generator\n",
    "np.random.seed(123)\n",
    "\n",
    "# True parameter values\n",
    "alpha, sigma = 1, 1; beta = [1, 2.5]\n",
    "\n",
    "# Predictor variable\n",
    "X1 = np.random.randn(100)\n",
    "X2 = np.random.randn(100) * 0.2\n",
    "\n",
    "# Simulate outcome variable\n",
    "Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(100)*sigma\n",
    "Y=Y.astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequential sampling (1 chains in 1 job)\n",
      "CompoundStep\n",
      ">Metropolis: [<tf.Variable 'alpha:0' shape=() dtype=float32_ref>]\n",
      ">Metropolis: [<tf.Variable 'beta:0' shape=(2,) dtype=float32_ref>]\n",
      "100%|██████████| 5500/5500 [00:12<00:00, 455.22it/s]\n",
      "Only one chain was sampled, this makes it impossible to run some convergence checks\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.905485\n",
      "[0.947812 2.62094 ]\n"
     ]
    }
   ],
   "source": [
    "basic_model2 = pm.Model()\n",
    "with basic_model2:\n",
    "    \n",
    "    # requires initial value now. \n",
    "    # also has to be float at the moment. \n",
    "    alpha = pm.Normal(name='alpha',initial_value=0.0, mu=0.0, sd=10.0) \n",
    "    beta = pm.Normal(name='beta',initial_value=np.array([1.0,1.0],dtype='float32'), mu=0.0, sd=10.0, shape=2)\n",
    "\n",
    "    mu = alpha + beta[0]*X1 + beta[1]*X2\n",
    "\n",
    "    Y_obs = pm.Normal(name='Y_obs',initial_value=np.ones_like(Y), mu=mu, sd=1.0, observed=Y)\n",
    "\n",
    "    step = pm.Metropolis()\n",
    "    trace = pm.sample(5000,step=step,cores=1,chains=1)\n",
    "\n",
    "print(np.mean(trace['alpha:0']))\n",
    "print(np.mean(trace['beta:0'],axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a bit slower, but it works! Notice the variable names are `tf.Variables` now. \n",
    "\n",
    "The parameter estimates are virtually the same, but not exactly. One reason for the slight differences may be that I set the TensorFlow variables to be float32's and Theano variables default to float64's. "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}