worksheet05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP90051 Workshop 5\n",
    "## Logistic regression in TensorFlow\n",
    "***\n",
    "In this worksheet, you'll get some hands-on experience with TensorFlow by implementing logistic regression using the low-level API.\n",
    "We'll assume you're already familiar with logistic regression from Worksheet 3, although we'll consider the more general case of multi-class classification.\n",
    "By the end of this worksheet, you should be able to:\n",
    "* use placeholders to feed data into TensorFlow\n",
    "* define a simple graph (e.g. to represent the logistic regression model)\n",
    "* use the built-in optimisers to fit a model\n",
    "\n",
    "Let's begin by importing the required packages (including TensorFlow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. MNIST data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST is a multi-class classification data set where:\n",
    "* the features are images of handwritten digits (28×28 pixels with a single 8-bit channel)\n",
    "* the target is a label in the set $\\{0, 1, \\ldots, 9\\}$\n",
    "\n",
    "The data is already split into training and test sets. The training set contains 60,000 instances and the test set contains 10,000 instances.\n",
    "\n",
    "Below we load the data into NumPy arrays using a built-in function from TensorFlow.\n",
    "\n",
    "**Question:** How are the arrays structured? Which index is used to access individual instances? What is the type of the arrays? \\[Hint: use `array.dtype` to check\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "(images_train, labels_train), (images_test, labels_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using the data for logistic regression, we need to do some basic pre-processing: \n",
    " * We rescale the images so that each pixel is represented as a float between 0 and 1 (note that the images are already normalised)\n",
    " * Unroll the 2D image arrays into 1D arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_train = np.reshape(images_train.astype('float32')/255, [images_train.shape[0], -1])\n",
    "features_test = np.reshape(images_test.astype('float32')/255, [images_test.shape[0], -1])\n",
    "NUM_FEATURES = features_train.shape[1]\n",
    "NUM_CLASSES = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore the training instances using the code block below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_id = 4 # change this\n",
    "plt.imshow(images_train[train_id], cmap=\"binary\")\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print(\"This instance has label {}\".format(labels_train[train_id]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Batching the data\n",
    "To fit the logistic regression model, we'll be using gradient descent with mini-batches.\n",
    "In other words, at each iteration we update the parameters using a small sample (mini-batch) of points from the training set.\n",
    "\n",
    "To make our code more readable, we implement a lightweight wrapper for the training data which returns mini-batches when requested.\n",
    "This functionality can be implemented elegantly using a Python iterator (see below).\n",
    "\n",
    "**Note:** For large-scale data (particularly when using a GPU) we recommend using the new Dataset API in Tensorflow (available at `tf.data.Dataset`). \n",
    "It can represents a generic input pipeline (which might include batching as well as transformations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DatasetIterator:\n",
    "    \"\"\"\n",
    "    An iterator that returns randomized batches from a data set (with features and labels)\n",
    "    \"\"\"\n",
    "    def __init__(self, features, labels, batch_size):\n",
    "        assert(features.shape[0]==labels.shape[0])\n",
    "        assert(batch_size > 0 and batch_size <= features.shape[0])\n",
    "        self.features = features\n",
    "        self.labels = labels\n",
    "        self.num_instances = features.shape[0]\n",
    "        self.batch_size = batch_size\n",
    "        self.num_batches = self.num_instances//self.batch_size\n",
    "        if (self.num_instances%self.batch_size!=0):\n",
    "            self.num_batches += 1\n",
    "        self._i = 0\n",
    "        self._rand_ids = None\n",
    "\n",
    "    def __iter__(self):\n",
    "        self._i = 0\n",
    "        self._rand_ids = np.random.permutation(self.num_instances)\n",
    "        return self\n",
    "        \n",
    "    def __next__(self):\n",
    "        if self.num_instances - self._i >= self.batch_size:\n",
    "            this_rand_ids = self._rand_ids[self._i:self._i + self.batch_size]\n",
    "            self._i += self.batch_size\n",
    "            return self.features[this_rand_ids], self.labels[this_rand_ids]\n",
    "        elif self.num_instances - self._i > 0:\n",
    "            this_rand_ids = self._rand_ids[self._i::]\n",
    "            self._i = self.num_instances\n",
    "            return self.features[this_rand_ids], self.labels[this_rand_ids]\n",
    "        else:\n",
    "            raise StopIteration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define a batch size and set up an iterator for the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "train_iterator = DatasetIterator(features_train, labels_train, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. TensorFlow placeholders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Placeholders represent tensors in the dataflow graph that will be fed values from outside TensorFlow (e.g. NumPy arrays).\n",
    "\n",
    "Below we define some placeholders for features and labels.\n",
    "Note that we've used `None` for the first dimension of both placeholders.\n",
    "This indicates that the size of the first dimension (i.e. the number of instances) is not fixed.\n",
    "This allows us to feed in data with varying numbers of instances (e.g. a batch might only have 100 instances, while the test set has 10,000 instances)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.placeholder(dtype=tf.float32, shape=[None, NUM_FEATURES], name=\"features\")\n",
    "Y = tf.placeholder(dtype=tf.uint8, shape=[None,], name=\"labels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll shortly specify the logistic regression model as part of the dataflow graph.\n",
    "However, first we consider how to generalize binary logistic regression to multi-class softmax logistic regression.\n",
    "***\n",
    "It's convenient to encode the multi-class labels (corresponding to the digits 0-9) as one-hot vectors.\n",
    "A one-hot vector has length equal to the number of classes. Whichever class is \"active\" is set to $1$, while the remaining classes are set to $0$. \n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "y = 0 &\\to \\mathbf{y} = \\begin{bmatrix}\n",
    "1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\n",
    "\\end{bmatrix} \\\\\n",
    "y = 1 &\\to \\mathbf{y} = \\begin{bmatrix}\n",
    "0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\n",
    "\\end{bmatrix} \\\\\n",
    "&\\vdots \\\\\n",
    "y = 9 &\\to \\mathbf{y} = \\begin{bmatrix}\n",
    "0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\n",
    "\\end{bmatrix}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "For training instance $(\\mathbf{x}_i, \\mathbf{y}_i)$:\n",
    "\n",
    "* Define $\\mathbf{z}_i = \\mathbf{x}_i  \\mathbf{W} + \\mathbf{b}$ where $\\mathbf{W}$ is a `num_features` × `num_classes` weight matrix and $\\mathbf{b}$ is a bias vector of length `num_classes`.\n",
    "* The predicted label is $\\hat{y}_i = \\arg \\max_{j} \\sigma(\\mathbf{z}_i)_{j}$ where $\\sigma(\\mathbf{z})_j = \\frac {e^{z_{j}}}{\\sum _{k=1}^{K} e^{z_{k}}}$ is the softmax function.\n",
    "\n",
    "The corresponding loss function is the softmax cross-entropy:\n",
    "$$\n",
    "L[\\mathbf{W}, \\mathbf{b}] = - \\sum_{i} \\mathbf{y}_i \\log (\\sigma(\\mathbf{z}_i))\n",
    "$$\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To implement this model in TensorFlow, we need to define Tensor variables to store the weight matrix and the bias vector.\n",
    "We do this using `tf.get_variable` (although you may see `tf.Variable` used elsewhere).\n",
    "Note that we had to specify the shape of the tensors, the type, and an initializer (we initialise to zero here).\n",
    "\n",
    "Once we've defined the variables, we can define additional tensors using TensorFlow operations (e.g. `tf.matmul` for matrix multiplication).\n",
    "Ultimately, we need to tell TensorFlow how to compute the loss, so that we can minimize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope(\"logistic-regression\", reuse=tf.AUTO_REUSE):\n",
    "    W = tf.get_variable(\"weights\", shape=[NUM_FEATURES, NUM_CLASSES], \\\n",
    "                        dtype=tf.float32, initializer=tf.zeros_initializer())\n",
    "    b = tf.get_variable(\"bias\", shape=[NUM_CLASSES], dtype=tf.float32, \\\n",
    "                        initializer=tf.zeros_initializer())\n",
    "    logits = tf.matmul(X, W) + b\n",
    "    Y_pred = tf.nn.softmax(logits)\n",
    "    Y_one_hot = tf.one_hot(Y, NUM_CLASSES)\n",
    "    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels=Y_one_hot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Running gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've told TensorFlow how to compute the loss (by defining the `loss` tensor) we can apply a built-in optimizer.\n",
    "\n",
    "Note that we don't have to specify how to compute the gradient of the loss. This is one of the advantages of expressing the computation as a data flow graph—we can get derivatives using automatic differentiation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = tf.train.GradientDescentOptimizer(0.01)\n",
    "opt_operation = opt.minimize(loss)\n",
    "num_epochs = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we set up an initializer for the global variables (including `W` and `b`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And open a session to run operations on the graph we've defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    #writer = tf.summary.FileWriter(\"./tensorboard.log\", sess.graph) # uncomment to use TensorBoard\n",
    "    \n",
    "    # Run gradient descent for multiple epochs\n",
    "    for epoch in range(num_epochs):\n",
    "        avg_loss = 0 # average loss over all batches\n",
    "        for X_batch, Y_batch in train_iterator:\n",
    "            _, l = sess.run([opt_operation, loss], feed_dict={X: X_batch, Y: Y_batch})\n",
    "            avg_loss += l / train_iterator.num_batches\n",
    "        print(\"Epoch {}: loss={:.9f}\".format(epoch, avg_loss))\n",
    "    print(\"Optimization complete.\")\n",
    "    \n",
    "    # Evaluate the trained model\n",
    "    correct_prediction = tf.equal(tf.argmax(Y_pred, 1), tf.argmax(Y_one_hot, 1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    print(\"Accuracy on the test set: {}.\".format(accuracy.eval({X: features_test, Y: labels_test})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Extension activities\n",
    "* Experiment with using different built-in optimizers (e.g. `tf.train.AdamOptimizer`) and/or step sizes. Does the rate of convergence vary?\n",
    "* Try setting up TensorBoard (a web-app that allows you to view the graph and visualize metrics during training).\n",
    "* Adapt the code in this notebook to implement linear regression (it should only require minimal changes). You'll need to import data that's appropriate for regression."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}