02.1_code_a_perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Practicum AI Logo image](images/practicum_ai_logo.png) <img src='images/practicumai_deep_learning.png' alt='Practicum AI: Deep Learning Foundations icon' align='right' width=50>\n",
    "\n",
    "\n",
    "***\n",
    "# *Practicum AI:* Deep Learning - Perceptron\n",
    "\n",
    "\n",
    "> This exercise adapted from the [W3 Schools Perceptrons](https://www.w3schools.com/ai/ai_perceptrons.asp) article and from Baig et al. (2020) The Deep Learning Workshop from [Packt Publishers](https://www.packtpub.com/product/the-deep-learning-workshop/9781839219856) (Exercise 2.01, page 55).\n",
    "\n",
    "<img alt=\"A cartoon of Dr. Amelia, a nutrition researcher, sitting at a computer thinking about food items which appear in a thought bubble.\" src=\"images/DrAmelia.jpg\" align=\"right\" width=250>Amelia is back! This time, she needs your help to analyze some of her survey data. As part of Amelia's dietary study, participants are also asked to follow a special nutrition plan, the Dr. Amelia Recommended Nutrition Plan (the DARN Plan). We'll use a simple [perceptron](https://developers.google.com/machine-learning/glossary#perceptron) to predict if participants follow the DARN Plan. \n",
    "\n",
    "**Note:** Dr. Amelia's cartoon was generated with AI's assistance.\n",
    " \n",
    "As a note, this exercise lies somewhere between coding everything from scratch and relying on the pre-coded APIs (Application Programming Interfaces) that underlie the power of TensorFlow, Keras, and Pytorch. **You will not need to create weight tensors beyond this exercise**. Still, hopefully, by doing it this time, you will have a better understanding (*and appreciation*) of the details often lost in an API call to `model.fit()`, for example.\n",
    "\n",
    "The table below shows some data Amelia has gathered from participant surveys about their nutrition. She is looking at how different factors predict if participants follow her DARN Plan ($y$, the output or [labels](https://developers.google.com/machine-learning/glossary#label) in our example) based on three input variables: if participants submit photos of three meals a day ($x_1$), if participants report being satisfied with their food choices ($x_2$), and if participants report being generally happy ($x_3$). We will combine $x_1$, $x_2$, and $x_3$ into our input tensor $X$. Here, we are simplifying the question of the likelihood of following the DARN Plan to a Yes/No. \n",
    "\n",
    "Case # | Photos of 3 meals submitted? ($x_1$) | Satisfied with food choices? ($x_2$) | Generally happy? ($x_3$) | Following the DARN Plan? ($y$)\n",
    "--|--------------------------|---------------------|-----------------------|----------------\n",
    "1 | 1 (Yes) | 1 (Yes) | 1 (Yes) | Yes (1)\n",
    "2 | 0 (No) | 1 (Yes) | 1 (Yes) | Yes (1)\n",
    "3 | 1 (Yes) | 0 (No) | 1 (Yes) | Yes (1)\n",
    "4 | 0 (No) | 0 (No) | 1 (Yes) | Yes (1)\n",
    "5 | 1 (Yes) | 1 (Yes) | 0 (No) | Yes (1)\n",
    "6 | 0 (No) | 1 (Yes) | 0 (No) | No (0)\n",
    "7 | 1 (Yes) | 0 (No) | 0 (No) | No (0)\n",
    "8 | 0 (No) | 0 (No) | 0 (No) | No (0)\n",
    "\n",
    "\n",
    "## 1. Import libraries\n",
    "\n",
    "### <img src='images/note_icon.svg' width=40, align='center' alt='Note icon'> Note\n",
    "\n",
    "> We'll probably stop reminding you after this, but... remember not all red output is bad!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-08 15:01:30.484317: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-01-08 15:01:30.484364: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-01-08 15:01:30.485438: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-01-08 15:01:30.491089: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-01-08 15:01:31.167420: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "from tensorflow.keras import activations\n",
    "\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an input data matrix\n",
    "\n",
    "Create a 3 x 8 matrix for our input data. Remember that we have three input variables (we'll call them $x_1$, $x_2$, and $x_3$ for now). These variables are the columns in our input data.\n",
    "\n",
    "The matrix below has the three input columns of our data table, using just the 0/1 values corresponding to the no/yes entries in the table. The comments help match rows of the table with entries in our `X` variable. (Remember, we are using the capital letter `X` as our variable name here to remind us that this is a matrix with our input data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(8, 3) dtype=float32, numpy=\n",
      "array([[1., 1., 1.],\n",
      "       [0., 1., 1.],\n",
      "       [1., 0., 1.],\n",
      "       [0., 0., 1.],\n",
      "       [1., 1., 0.],\n",
      "       [0., 1., 0.],\n",
      "       [1., 0., 0.],\n",
      "       [0., 0., 0.]], dtype=float32)>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-08 15:01:36.848132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7918 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-80GB MIG 1g.10gb, pci bus id: 0000:c8:00.0, compute capability: 8.0\n"
     ]
    }
   ],
   "source": [
    "X = tf.Variable([[1.,1.,1.], # Case 1\n",
    "                 [0.,1.,1.], # Case 2\n",
    "                 [1.,0.,1.], # Case 3\n",
    "                 [0.,0.,1.], # Case 4\n",
    "                 [1.,1.,0.], # Case 5\n",
    "                 [0.,1.,0.], # Case 6\n",
    "                 [1.,0.,0.], # Case 7\n",
    "                 [0.,0.,0.]], # Case 8\n",
    "                 dtype = tf.float32)  # 3x8, input data table\n",
    "print(X)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create a label tensor\n",
    "\n",
    "Create a tensor of labels to hold our 'ground truth'. This indicates, for each set of input, whether or not the participant is following the DARN Plan. \n",
    "\n",
    "```python\n",
    "# Outputs:       1, 2, 3, 4, 5, 6, 7, 8 - one for each case in the table         \n",
    "y = tf.Variable([1, 1, 1, 1, 1, 0, 0, 0], dtype = tf.float32) \n",
    "\n",
    "# Reshape to be 8 rows of 1 column  \n",
    "y = tf.reshape(y, [8,1]) \n",
    "print(y)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[1.]\n",
      " [1.]\n",
      " [1.]\n",
      " [1.]\n",
      " [1.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]], shape=(8, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "y = tf.Variable([1, 1, 1, 1, 1, 0, 0, 0], dtype = tf.float32) \n",
    "\n",
    "# Reshape to be 8 rows of 1 column  \n",
    "y = tf.reshape(y, [8,1]) \n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Define some constants to set the shape of the weight matrix\n",
    "\n",
    "Define two constants to be used in the next step when we define the connections weight matrix.\n",
    "\n",
    "We can use the number of columns in the X table to determine the number of features or how many $x_i$ we have and, therefore, how many weights we need to store (one for each feature). We only need one output value since we are looking for a binary decision about plan adherence (Yes/No)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "num_features = X.shape[1]\n",
    "output_size = 1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code it!\n",
    "num_features = X.shape[1]\n",
    "output_size = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## 5. Define connections weight matrix\n",
    "\n",
    "![Diagram of the perceptron with 3 input variables (x1, x2, x3), 3 weights (w1, W2, w3) and the bias term. The perceptron body multiplies the inputs by the weights and sums them and the bias, resulting in the output--whether or not the participant is following the DARN Plan. The three weights are highlighted here.](images/02_perceptron_section5.png)\n",
    "\n",
    "In our feature matrix, we will need one weight for each feature, $x_i$ (three photos submitted, satisfied with food choices, etc.), labeled $X$. These weights are our $w_i$. We don't know what value they should take so we will initialize them to 0. Another common option is to use a random number to initialize the weights - this is one reason different runs of model training may give different answers.\n",
    "\n",
    "```python\n",
    "W = tf.Variable(tf.zeros([num_features, output_size]), dtype = tf.float32)\n",
    "print(W)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\n",
      "array([[0.],\n",
      "       [0.],\n",
      "       [0.]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "W = tf.Variable(tf.zeros([num_features, output_size]), dtype = tf.float32)\n",
    "print(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## 6. Define bias variable\n",
    "\n",
    "![Diagram of the perceptron with 3 input variables (x1, x2, x3), 3 weights (w1, W2, w3) and the bias term. This is similar to the above image, but is highlighting the bias term](images/02_perceptron_section6.png)\n",
    "\n",
    "Since we only have one neuron, we only need one bias value. Again, we'll initialize it to 0 - a random number would be another option here. We can write each bias term as $b_i$ and the matrix of all biases as $B$.\n",
    "\n",
    "```python\n",
    "B = tf.Variable(tf.zeros([output_size, 1]), dtype = tf.float32)\n",
    "print(B)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[0.]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "B = tf.Variable(tf.zeros([output_size, 1]), dtype = tf.float32)\n",
    "print(B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## 7. Define a perceptron function\n",
    "\n",
    "![Diagram of the perceptron with 3 input variables (x1, x2, x3), 3 weights (w1, W2, w3) and the bias term. This is similar to the above image, but is highlighting the perceptron body.](images/02_perceptron_section7.png)\n",
    "\n",
    "\n",
    "In the following code block, we define a perceptron function with one input argument, $X$, containing our three input data features. \n",
    "\n",
    "The function's first line implements a net input function.  It multiplies the input data matrix ($X$) by the weights ($W$) using the matrix multiplication function (matmul).  It then adds the bias ($B$) value to that product.\n",
    "\n",
    "### <img src='images/note_icon.svg' width=40, align='center' alt='Note icon'>Note\n",
    "> This is the essential function of a neuron: gather the inputs, multiply each input by the weight for that input, add the products up and add in the bias.\n",
    "\n",
    "The function's second line implements an activation function. The activation function determines how the neuron's output (calculated above) is changed before passing it on. Here, we use the `tanh` activation function.  However, there are other TensorFlow options.  For example, you could use the `tf.sigmoid` function.  Or, select a function from the Keras activation (`activations`) library.  Search the [Keras documentation](https://keras.io/api/layers/activations/) for a complete list of available functions.\n",
    "\n",
    "Try out these other options, retrain the network, and see what happens.\n",
    "\n",
    "```python\n",
    "output = tf.sigmoid(z)\n",
    "output = activations.relu(z)\n",
    "output = activations.linear(z)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perceptron(X):\n",
    "    z = tf.add(tf.matmul(X, W), B)  # Net input function\n",
    "    output = tf.tanh(z)             # Activation function\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the perceptron function to see its initial predictions before any training.  All of its predictions ought to be 0 (remember we set all the weights and the bias to 0 - so whatever the inputs are, they are all multiplied by 0 and have 0 added to the sum). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]], shape=(8, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Execute the perceptron to see its initial predictions before training.\n",
    "print(perceptron(X))       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Training the Perceptron\n",
    "\n",
    "Now that we have the elements of a simple, single-node perceptron in place, let's train the network using an algorithm called \"stochastic gradient descent\" (SGD). The purpose of SGD is to iteratively adjust the weights and bias parameters of the single neuron in our model and eventually, we hope, find values that make our neuron's predictions as good as possible. Tensorflow/Keras implements this algorithm for us, so we don't need to code it ourselves.\n",
    "\n",
    "The [learning rate](https://developers.google.com/machine-learning/glossary#learning-rate) determines the size of the steps taken towards the global minimum. Here, the Stochastic Gradient Descent (SGD) optimizer has been selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "optimizer = tf.optimizers.SGD(learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Train the perceptron for 1000 epochs\n",
    "\n",
    "An [epoch](https://developers.google.com/machine-learning/glossary#epoch) is a complete training pass over the entire dataset. Our loss or error function is defined as a lambda function (a single-line, inline function) in the first line of code in the loop block.  We use the `sigmoid_cross_entropy_with_logits` function, an appropriate choice for this application, to calculate how far our predicted results are from the known results. We will not get into the technical details here as that is outside the scope of this learning experience. Our SGD optimizer seeks to minimize the model's total error in the second line.\n",
    "\n",
    "### <img src='images/note_icon.svg' width=40, align='center' alt='Note icon'>Note\n",
    "> The code below uses a `for` loop. This common programming construct allows you to loop, or iterate, through\n",
    "> a list of items (the numbers 0 to 999 in our case). *Implicitly*, training will use `for` loops - for each epoch do \n",
    "> this thing. *Explicitly*, however, after this notebook, we will use the API that automatically does this for us.\n",
    "> Thus we dropped coverage of `for` loops and other \"flow control\" methods from the *Python for AI* course. It's\n",
    "> helpful to know about them, but they are rarely used explicitly in AI research.\n",
    "> [Click here for more details](https://wiki.python.org/moin/ForLoop).\n",
    ">\n",
    "> The block also uses a special Python function called a `lambda` function. These are functions that can be \n",
    "> written as a single line of code. [Click here for more\n",
    "> details](https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-08 15:05:56.709746: I external/local_tsl/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory\n",
      "2024-01-08 15:05:57.147393: I external/local_xla/xla/service/service.cc:168] XLA service 0x5582df628950 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
      "2024-01-08 15:05:57.147426: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA A100-SXM4-80GB MIG 1g.10gb, Compute Capability 8.0\n",
      "2024-01-08 15:05:57.412140: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904\n",
      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
      "I0000 00:00:1704747957.459782 3175536 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.\n"
     ]
    }
   ],
   "source": [
    "no_of_epochs = 1000\n",
    "\n",
    "for n in range(no_of_epochs):\n",
    "    loss = lambda:abs(tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels = y, logits = perceptron(X))))\n",
    "    optimizer.minimize(loss, [W, B])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Print the weights\n",
    "<img alt=\"AI Generated cartoon of happy people eating healthy food.\" src=\"images/happy_people.jpg\" align=\"right\" width=\"300\">\n",
    "\n",
    "Notice that the model has learned that the general happiness of a participant is the best predictor of whether or not they are following the DARN Plan! Of the weights, the 3rd one has the largest value.\n",
    "\n",
    "Given that the input from each feature will be a 0 or a 1, multiplying by a larger weight will increase the contribution of that feature in the summation of all input-by-weight products ($x_i * w_i$) in determining the output of the neuron.\n",
    "\n",
    "The perceptron has learned how to take the three input variables and weigh them to predict the output. \n",
    "\n",
    "**Note:** The image was generated with AI's assistance.\n",
    "\n",
    "```python\n",
    "print(W)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\n",
      "array([[0.29068494],\n",
      "       [0.29068494],\n",
      "       [1.0723362 ]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "print(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Print the bias\n",
    "\n",
    "```python\n",
    "print(B)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[-0.3337661]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "print(B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Test the perceptron\n",
    "\n",
    "The numbers in the output tensor reflect the perceptron's predictions for each input case. These are not probabilities but the **model's estimate of the output value**. We could set a threshold value and conclude the participant is following the DARN Plan when the value exceeds some number.\n",
    "\n",
    "```python\n",
    "print(perceptron(X))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 0.8667691 ]\n",
      " [ 0.7736094 ]\n",
      " [ 0.7736094 ]\n",
      " [ 0.62828046]\n",
      " [ 0.24266483]\n",
      " [-0.04305453]\n",
      " [-0.04305453]\n",
      " [-0.3219007 ]], shape=(8, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Code it!\n",
    "print(perceptron(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print things more clearly\n",
    "\n",
    "Let's bring the `X`, `y` and predictions together to make it easier to read. Remember that `Yes=1` and `No=0` in the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Photos of 3 meals submitted?</th>\n",
       "      <th>Satisfied with food choices?</th>\n",
       "      <th>Generally happy?</th>\n",
       "      <th>Following the DARN Plan?</th>\n",
       "      <th>Predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.866769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.773609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.773609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.628280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.242665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.043055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.043055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.321901</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Photos of 3 meals submitted?  Satisfied with food choices?  \\\n",
       "0                           1.0                           1.0   \n",
       "1                           0.0                           1.0   \n",
       "2                           1.0                           0.0   \n",
       "3                           0.0                           0.0   \n",
       "4                           1.0                           1.0   \n",
       "5                           0.0                           1.0   \n",
       "6                           1.0                           0.0   \n",
       "7                           0.0                           0.0   \n",
       "\n",
       "   Generally happy?  Following the DARN Plan?  Predictions  \n",
       "0               1.0                       1.0     0.866769  \n",
       "1               1.0                       1.0     0.773609  \n",
       "2               1.0                       1.0     0.773609  \n",
       "3               1.0                       1.0     0.628280  \n",
       "4               0.0                       1.0     0.242665  \n",
       "5               0.0                       0.0    -0.043055  \n",
       "6               0.0                       0.0    -0.043055  \n",
       "7               0.0                       0.0    -0.321901  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_df = pd.DataFrame(X.numpy(), columns=['Photos of 3 meals submitted?', 'Satisfied with food choices?', 'Generally happy?'])\n",
    "y_df = pd.DataFrame(y.numpy(), columns=['Following the DARN Plan?'])\n",
    "pred_df = pd.DataFrame(perceptron(X).numpy(), columns=['Predictions'])\n",
    "df = pd.concat([X_df, y_df, pred_df], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Let's see how different choices would change the results\n",
    "\n",
    "Let's change the participant outcomes and see what happens to the learned weights and predictions. \n",
    "\n",
    "### Change 1: participants are more likely to follow the DARN Plan when they like the food choices:\n",
    "\n",
    "`y = tf.Variable([1, 1, 0, 0, 1, 0, 0, 0], dtype = tf.float32)`\n",
    "\n",
    "### Change 2: participants are more likely to follow the DARN Plan when they regularly submit three photos a day:\n",
    "\n",
    "`y = tf.Variable([1, 0, 1, 0, 1, 0, 1, 0], dtype = tf.float32)`\n",
    "\n",
    "Feel free to play with other parts of the model; everything but the X inputs is replicated below to put it all in one place for easy reference. Comments point out hyperparameters that you might want to change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights: <tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\n",
      "array([[0.15107682],\n",
      "       [0.9818896 ],\n",
      "       [0.15107682]], dtype=float32)>\n",
      "Bias: <tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[-0.794678]], dtype=float32)>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Photos of 3 meals submitted?</th>\n",
       "      <th>Satisfied with food choices?</th>\n",
       "      <th>Generally happy?</th>\n",
       "      <th>Following the DARN Plan?</th>\n",
       "      <th>Predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.453713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.325949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.456218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.567347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.325949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.185055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.567347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.661051</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Photos of 3 meals submitted?  Satisfied with food choices?  \\\n",
       "0                           1.0                           1.0   \n",
       "1                           0.0                           1.0   \n",
       "2                           1.0                           0.0   \n",
       "3                           0.0                           0.0   \n",
       "4                           1.0                           1.0   \n",
       "5                           0.0                           1.0   \n",
       "6                           1.0                           0.0   \n",
       "7                           0.0                           0.0   \n",
       "\n",
       "   Generally happy?  Following the DARN Plan?  Predictions  \n",
       "0               1.0                       1.0     0.453713  \n",
       "1               1.0                       1.0     0.325949  \n",
       "2               1.0                       0.0    -0.456218  \n",
       "3               1.0                       0.0    -0.567347  \n",
       "4               0.0                       1.0     0.325949  \n",
       "5               0.0                       0.0     0.185055  \n",
       "6               0.0                       0.0    -0.567347  \n",
       "7               0.0                       0.0    -0.661051  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## From step 3\n",
    "# Outputs:       1, 2, 3, 4, 5, 6, 7, 8 - one for each case in the table         \n",
    "y = tf.Variable([1, 1, 0, 0, 1, 0, 0, 0], dtype = tf.float32)  # Change 1 has been made, you'll need to make change 2\n",
    "y = tf.reshape(y, [8,1])  # convert to 4x1\n",
    "\n",
    "## From step 4\n",
    "num_features = X.shape[1]\n",
    "output_size = 1\n",
    "\n",
    "## From step 5\n",
    "W = tf.Variable(tf.zeros([num_features, output_size]), dtype = tf.float32)\n",
    "\n",
    "## From step 6\n",
    "B = tf.Variable(tf.zeros([output_size, 1]), dtype = tf.float32)\n",
    "\n",
    "## From step 7\n",
    "def perceptron(X):\n",
    "    z = tf.add(tf.matmul(X, W), B)      \n",
    "    output = tf.tanh(z)                  # Activation function is a good hyperparameter to change \n",
    "    return output\n",
    "\n",
    "## From step 8\n",
    "learning_rate = 0.01  # Learning rate is a good hyperparameter to change\n",
    "optimizer = tf.optimizers.SGD(learning_rate)\n",
    "\n",
    "## From step 9\n",
    "no_of_epochs = 1000  # Number of epochs is a good hyperparameter to change\n",
    "\n",
    "for n in range(no_of_epochs):\n",
    "    loss = lambda:abs(tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels = y, logits = perceptron(X))))\n",
    "    optimizer.minimize(loss, [W, B])\n",
    "    \n",
    "## From steps 10 on, printing the output\n",
    "print(f'Weights: {W}')\n",
    "print(f'Bias: {B}')\n",
    "\n",
    "X_df = pd.DataFrame(X.numpy(), columns=['Photos of 3 meals submitted?', 'Satisfied with food choices?', 'Generally happy?'])\n",
    "y_df = pd.DataFrame(y.numpy(), columns=['Following the DARN Plan?'])\n",
    "pred_df = pd.DataFrame(perceptron(X).numpy(), columns=['Predictions'])\n",
    "df = pd.concat([X_df, y_df, pred_df], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights: <tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\n",
      "array([[ 1.4559333 ],\n",
      "       [-0.15017778],\n",
      "       [-0.15017778]], dtype=float32)>\n",
      "Bias: <tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[-0.46442914]], dtype=float32)>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Photos of 3 meals submitted?</th>\n",
       "      <th>Satisfied with food choices?</th>\n",
       "      <th>Generally happy?</th>\n",
       "      <th>Following the DARN Plan?</th>\n",
       "      <th>Predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.598719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.643887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.686511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.547362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.686511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.547362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.758003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.433687</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Photos of 3 meals submitted?  Satisfied with food choices?  \\\n",
       "0                           1.0                           1.0   \n",
       "1                           0.0                           1.0   \n",
       "2                           1.0                           0.0   \n",
       "3                           0.0                           0.0   \n",
       "4                           1.0                           1.0   \n",
       "5                           0.0                           1.0   \n",
       "6                           1.0                           0.0   \n",
       "7                           0.0                           0.0   \n",
       "\n",
       "   Generally happy?  Following the DARN Plan?  Predictions  \n",
       "0               1.0                       1.0     0.598719  \n",
       "1               1.0                       0.0    -0.643887  \n",
       "2               1.0                       1.0     0.686511  \n",
       "3               1.0                       0.0    -0.547362  \n",
       "4               0.0                       1.0     0.686511  \n",
       "5               0.0                       0.0    -0.547362  \n",
       "6               0.0                       1.0     0.758003  \n",
       "7               0.0                       0.0    -0.433687  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## From step 3\n",
    "# Outputs:       1, 2, 3, 4, 5, 6, 7, 8 - one for each case in the table         \n",
    "y = tf.Variable([1, 0, 1, 0, 1, 0, 1, 0], dtype = tf.float32)  # Change 1 has been made, you'll need to make change 2\n",
    "y = tf.reshape(y, [8,1])  # convert to 4x1\n",
    "\n",
    "## From step 4\n",
    "num_features = X.shape[1]\n",
    "output_size = 1\n",
    "\n",
    "## From step 5\n",
    "W = tf.Variable(tf.zeros([num_features, output_size]), dtype = tf.float32)\n",
    "\n",
    "## From step 6\n",
    "B = tf.Variable(tf.zeros([output_size, 1]), dtype = tf.float32)\n",
    "\n",
    "## From step 7\n",
    "def perceptron(X):\n",
    "    z = tf.add(tf.matmul(X, W), B)      \n",
    "    output = tf.tanh(z)                  # Activation function is a good hyperparameter to change \n",
    "    return output\n",
    "\n",
    "## From step 8\n",
    "learning_rate = 0.01  # Learning rate is a good hyperparameter to change\n",
    "optimizer = tf.optimizers.SGD(learning_rate)\n",
    "\n",
    "## From step 9\n",
    "no_of_epochs = 1000  # Number of epochs is a good hyperparameter to change\n",
    "\n",
    "for n in range(no_of_epochs):\n",
    "    loss = lambda:abs(tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels = y, logits = perceptron(X))))\n",
    "    optimizer.minimize(loss, [W, B])\n",
    "    \n",
    "## From steps 10 on, printing the output\n",
    "print(f'Weights: {W}')\n",
    "print(f'Bias: {B}')\n",
    "\n",
    "X_df = pd.DataFrame(X.numpy(), columns=['Photos of 3 meals submitted?', 'Satisfied with food choices?', 'Generally happy?'])\n",
    "y_df = pd.DataFrame(y.numpy(), columns=['Following the DARN Plan?'])\n",
    "pred_df = pd.DataFrame(perceptron(X).numpy(), columns=['Predictions'])\n",
    "df = pd.concat([X_df, y_df, pred_df], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before continuing\n",
    "###  <img src='images/alert_icon.svg' alt=\"Alert icon\" width=40 align=center> Alert!\n",
    "> Before continuing to another notebook within the same Jupyter session,\n",
    "> use the **\"Running Terminals and Kernels\" tab** (below the File Browser tab) to **shut down this kernel**. \n",
    "> This will free up this notebook's GPU memory, making it available for\n",
    "> your next notebook.\n",
    ">\n",
    "> Every time you run multiple notebooks within a Jupyter session with\n",
    "> a GPU, this should be done."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deep_learning_course",
   "language": "python",
   "name": "deep_learning_course"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}