Chapter 08 - Modern Convolutional Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bd3c7490",
   "metadata": {},
   "source": [
    "# Chapter 8: Modern Convolutional Neural Networks\n",
    "\n",
    "### Dhuvi karthikeyan\n",
    "\n",
    "02/15/2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "664867b7",
   "metadata": {},
   "source": [
    "## 8.1 Deep CNNs (AlexNet)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a98a05c",
   "metadata": {},
   "source": [
    "### 8.1.1 Representation Learning\n",
    "\n",
    "Pre 2012 era in computer vision:\n",
    "\n",
    "* Image representation was calculated mechanistically with geometrically inspired inductive biases:\n",
    "    * Bag of Visual Words (2003)\n",
    "    * SIFT: Scale Invariant Feature Transform (2004)\n",
    "    * HOG: Histograms of oriented gradient (2005)\n",
    "    * SURF: Speeded Up Robust Features (2006)\n",
    "    \n",
    "* Contrarians:\n",
    "    * Yann LeCun\n",
    "    * Yoshua Bengio\n",
    "    * Andrew Ng\n",
    "    * Shun-ichi Amari\n",
    "    * Geoff Hinton\n",
    "    * Juergen Schmidhuber\n",
    "    \n",
    "All thought the features should be learned along with what to do with them. The features should be hierarchically learned such that the richness is encapsulated in it. \n",
    "\n",
    "**AlexNet:** 2012, vs LeNet in 1995 is explained by a dearth of data and compute. ImageNet (1e6 images) was a competition that had only recently come out."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8ab5a5",
   "metadata": {},
   "source": [
    "### 8.1.2 AlexNet\n",
    "\n",
    "By winning the ImageNet Large Scale Visual Recognition Challenge (2012) by a large margin, AlexNet was able to shift the zeitgeist from cleverly constructed featurization to automatic learning of features by deep networks.\n",
    "\n",
    "#### Architecture\n",
    "\n",
    "Image -> 11x11 Conv (96) -> MaxPool(3) -> 5x5 Conv (256), MaxPool(3) -> [3x3 Conv(384)]*3, MaxPool(3) -> FC(4096) -> FC(4096) -> FC(1000)\n",
    "\n",
    "#### Activation Functions\n",
    "\n",
    "ReLU function helped in many cases (training stability and efficiency)\n",
    "\n",
    "#### Capacity Control and Preprocessing\n",
    "\n",
    "Dropout in the FC layers controls for complexity of the model and overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eab074bc",
   "metadata": {},
   "source": [
    "## 8.2 Networks Using Blocks (VGG)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40bde992",
   "metadata": {},
   "source": [
    "Visual Geometry Group @OXford first came up with the idea of using blocks as a building unit of neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b22ac95",
   "metadata": {},
   "source": [
    "### 8.2.1 VGG Blocks\n",
    "\n",
    "The functional unit of a CONV layer block consists of:\n",
    "    1. Convolutional layer + padding (to maintain activation dimensions)\n",
    "    2. ReLU or similar activation function\n",
    "    3. Pooling-layer for downsampling (distillation?)\n",
    "\n",
    "**The above imposes a limit of log_k(d) where k is the max pooling dimension on the number of layers that can be added before the dimension of the activation is 1x1.**\n",
    "\n",
    "As such one of the key breakthroughs was the idea of using multiple conv layers between pooling operations.\n",
    "\n",
    "Simonyan and Zisserman (2014) were interested in figuring out whether shallow wide Convnets vs Long Narrow ones were better. \n",
    "    \n",
    "    * Parameter-wise the number of params is 5*5*In_channels*Out_channels vs      3*3*In_channels*Out_channels\n",
    "    * Conv -> Conv with 3x3 window reaches same number of pixels as one 5x5 window with 1/3 fewer params\n",
    "\n",
    "Deeper and narrower = better\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206f37c1",
   "metadata": {},
   "source": [
    "### 8.2.2 VGG Network\n",
    "\n",
    "VGG applies the pooling at the end of each block and has 5 blocks which downsamples the image by two every time from 224x224 to 112, 56, 28, 14."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c6652da",
   "metadata": {},
   "source": [
    "## 8.3 Network in Network\n",
    "\n",
    "Network in Network blocks (2013) sought to address the challenge of having a large nunmber of parameters for the fully connected layers at the end of a convnet. \n",
    "    \n",
    "    * Unable to move the FC layer up without destroying the spatial structure of the features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6f647b7",
   "metadata": {},
   "source": [
    "### 8.3.1 NiN Blocks\n",
    "\n",
    "* Uses a single CONV layer followed by 2 sequential 1x1 conv layers (FC layer at each pixel location)\n",
    "\n",
    "* Kernel sizes and the output channels are the same as AlexNet\n",
    "\n",
    "* Number of output channels is reduced to nunmber of classes by a nin_block\n",
    "\n",
    "* Global average pooling layer, vector of logits [No Fully Connected Layer]\n",
    "\n",
    "Interestingly taking the average didn't harm the accuracy. Averaging across low-res images adds translational invariance as well"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93deb544",
   "metadata": {},
   "source": [
    "## 8.4 Multi-Branch Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e6ad5eb",
   "metadata": {},
   "source": [
    "GoogLeNet won the (2014) ImageNet Challenge with a network that distilled a stem, body, and head in CNNs. \n",
    "\n",
    "The design:\n",
    "    \n",
    "    * Stem: First 2-3 convolutions that extract low-level features from images\n",
    "    * Body: Convolutional blocks that process information\n",
    "    * Head: Maps features learned for the dowstream target task\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099b4759",
   "metadata": {},
   "source": [
    "### 8.4.1 Inception Block\n",
    "\n",
    "Four parallel branches of information flow that input is fed into:\n",
    "    1. 1x1 conv -> Concat layer\n",
    "    2. 1x1 conv -> 3x3 conv -> Concat layer\n",
    "    3. 1x1 conv -> 5x5 conv -> Concat layer\n",
    "    4. 3x3 MaxPool -> 1x1 conv -> Concat layer\n",
    "    \n",
    "* Explores the images with different spatial filters\n",
    "* Allocation of diff number of parameters for diff filters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f327fea2",
   "metadata": {},
   "source": [
    "### 8.4.2 GoogLeNet Model\n",
    "\n",
    "What is going on here with the number of channels accounting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96d43571",
   "metadata": {},
   "source": [
    "## 8.5 Batch Normalization\n",
    "\n",
    "* Pre-processing (Rescaling)\n",
    "* Numerical Stability\n",
    "* Regularization\n",
    "\n",
    "**All in one**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a8777b",
   "metadata": {},
   "source": [
    "### 8.5.1 Training Deep Networks\n",
    "\n",
    "Standardization of input features in standard machine learning helped model convergence by placing all the features (and thus parameters) on the same scale:\n",
    "\n",
    "    * Want rescaling such that theres unity across the diagonal\n",
    "    * Mean zero and variance sums to one\n",
    "\n",
    "In MLPs (and CNNs) the intermediate representations of the input experience distribution drift where along the layers and within one layer the variables can take on values of different magnitudes. Normalization of these could help with model convergence.\n",
    "\n",
    "\n",
    "* This framework helps explain why batch norm and layer norm have been so successful in extending the depth in which we can train neural networks \n",
    "\n",
    "\n",
    "Since deep nets are prone to overfitting, regularization via noise injections, is also a direct fallout of batch norm. \n",
    "\n",
    "$$ BN(x) = \\gamma * \\frac{x-\\hat{\\mu}}{\\hat{\\sigma}} + \\beta $$\n",
    "\n",
    "Because of the nature of the batch sizes and the proportionality to noise, 50-100 is bsz that works best in terms of the noise stability. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcbcabd9",
   "metadata": {},
   "source": [
    "### 8.5.2 Batch Norm Layers\n",
    "\n",
    "Implementation between FC and convolutions are slightly diff\n",
    "\n",
    "#### Fully Connected Layers\n",
    "\n",
    "h = $\\phi (BN(Wx+b))$\n",
    "\n",
    "#### Convolutional Layers\n",
    "\n",
    "Batch norm is applied on a per channel basis across all the locations to preserve invariance.\n",
    "\n",
    "#### Layer Normalization\n",
    "\n",
    "Batch norm but applied to one obs @ a time:\n",
    "\n",
    "$$ x -> LN(x) = \\frac{x - \\hat{\\mu}}{\\hat{\\sigma}}$$\n",
    "\n",
    "This applies the mean and var across the single input and normalizes it. It empirically prevents divergence by applying a standardization that is deterministic \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "46cda549",
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_norm(X, gamma, beta, moving_mean, moving_var, eps, momentum):\n",
    "    # Use is_grad_enabled to determine whether we are in training mode\n",
    "    if not torch.is_grad_enabled():\n",
    "        # In prediction mode, use mean and variance obtained by moving average\n",
    "        X_hat = (X - moving_mean) / torch.sqrt(moving_var + eps)\n",
    "    else:\n",
    "        assert len(X.shape) in (2, 4)\n",
    "        if len(X.shape) == 2:\n",
    "            # When using a fully connected layer, calculate the mean and\n",
    "            # variance on the feature dimension\n",
    "            mean = X.mean(dim=0)\n",
    "            var = ((X - mean) ** 2).mean(dim=0)\n",
    "        else:\n",
    "            # When using a two-dimensional convolutional layer, calculate the\n",
    "            # mean and variance on the channel dimension (axis=1). Here we\n",
    "            # need to maintain the shape of X, so that the broadcasting\n",
    "            # operation can be carried out later\n",
    "            mean = X.mean(dim=(0, 2, 3), keepdim=True)\n",
    "            var = ((X - mean) ** 2).mean(dim=(0, 2, 3), keepdim=True)\n",
    "        # In training mode, the current mean and variance are used\n",
    "        X_hat = (X - mean) / torch.sqrt(var + eps)\n",
    "        # Update the mean and variance using moving average\n",
    "        moving_mean = (1.0 - momentum) * moving_mean + momentum * mean\n",
    "        moving_var = (1.0 - momentum) * moving_var + momentum * var\n",
    "    Y = gamma * X_hat + beta  # Scale and shift\n",
    "    return Y, moving_mean.data, moving_var.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5387f863",
   "metadata": {},
   "source": [
    "### 8.5.6 Discussion \n",
    "\n",
    "* Batch norm modulates hidden representations by taking minibatch statistics and transforming the values, boosting numerical stability\n",
    "* Batch norm implementation is different for diff network architectures\n",
    "* Batch norm has different functionality in training vs test\n",
    "* Noise-injection regularization peaks at mini-batches of around 50-100 training examples\n",
    "\n",
    "\n",
    "Batch norm makes the optimization landscape smoother?\n",
    "\n",
    "Internal covariate shift (debunked?) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84e2ddf",
   "metadata": {},
   "source": [
    "## 8.6 Residual Networks (ResNet)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7022dbf0",
   "metadata": {},
   "source": [
    "### 8.6.1 Function Classes\n",
    "\n",
    "Since neural networks are approximators of complex functions, the notion of a model architecture representing a particular class of functions is intuitive. Nested function classes are a desired property of adding complexity to NNs as they offer a formal framework by which increasing complexity necessarily leads to an optimization topology that contains regions closer to the true function than without the added layer. \n",
    "\n",
    "Mathmematically, one of the ways in which this is possible is if the added layer is able to be learn the identity function. He et al (2016) developed the ResNet which won 2015 ImageNet competition and introduced the residual block which baked in the idea that each additional layer should have the identity function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "363e875d",
   "metadata": {},
   "source": [
    "### 8.6.2 Residual Block"
   ]
  },
  {
   "cell_type": "raw",
   "id": "068e4cab",
   "metadata": {},
   "source": [
    "x -> 3x3 Conv -> BN -> ReLU -> 3x3 Conv -> BN + -> ReLU\n",
    "|                                             ^\n",
    "|                                             |\n",
    "V -------------------[1x1 Conv]---------------|"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed23641",
   "metadata": {},
   "source": [
    "### 8.6.3 ResNet Model\n",
    "\n",
    "Model starts off like GoogLeNet\n",
    "\n",
    "Blocks:\n",
    "\n",
    "* Block 1 (1x):\n",
    "    * 7x7 Conv\n",
    "    * BN\n",
    "    * 3x3 MaxPool\n",
    "    * ReLU\n",
    "* Block 2 (2x):\n",
    "    * ResBlock (No 1x1 conv) \n",
    "* Block 3 (3x):\n",
    "    * ResBlock (with 1x1 conv)\n",
    "    * ResBlock (w/o 1x1 conv)\n",
    "* Block 4:\n",
    "    * Global AvePool\n",
    "    * FC Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec7666e2",
   "metadata": {},
   "source": [
    "### 8.6.5 ResNeXt\n",
    "\n",
    "ResNet has an inherent tradeoff between non-linearity and dimensionality in a Residual Block.\n",
    "    \n",
    "    * Easily addressible by increasing the number of layers or the width of the conv filter\n",
    "    * Increase the number of channels (comes at quadratically increasing cost)\n",
    "    \n",
    "Branching a residual block into having g groups or branches and having them be equivalent architectures, thus sharing b intermediate dimensions over g groups or b/g channels per group.\n",
    "\n",
    "Computational cost goes from O(c_i * c_o) to O(g * (c_i/g/) * g(c_o/g)) = O(c_i * (c_o/g)) which is g times faster and holds g times less params. \n",
    "\n",
    "**Note** Must ensure no information leakage between groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60fb75c1",
   "metadata": {},
   "source": [
    "### 8.6.6 Summary and Discussion\n",
    "\n",
    "Nested function classes result in strictly more expressive and therefore powerful models. Allowing input to pass through allows this, changing inductive bias from f(x) = 0 to f(x) = x."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5225ad0c",
   "metadata": {},
   "source": [
    "## 8.7 Densely Connected Networks (DenseNet)\n",
    "\n",
    "DenseNet - Dense Convolutional Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f624a636",
   "metadata": {},
   "source": [
    "### 8.7.1 ResNet to DenseNet \n",
    "\n",
    "Densenet takes the idea of Resnet and asks what if instead of adding the input to the residual, we could instead apply a higher dimensional feature aggregation and uses the concatenation operator.\n",
    "\n",
    "$$ x --> [x, f_1(x), f_2([x, f_1(x)]), f_3([x, f_1(x),f_2([x, f_1(x)])])] -> MLP\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac96e1ab",
   "metadata": {},
   "source": [
    "### 8.7.2 Dense Blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "684a617d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseBlock(nn.Module):\n",
    "    def __init__(self, num_convs, num_channels):\n",
    "        super(DenseBlock, self).__init__()\n",
    "        layer = []\n",
    "        for i in range(num_convs):\n",
    "            layer.append(conv_block(num_channels))\n",
    "        self.net = nn.Sequential(*layer)\n",
    "\n",
    "    def forward(self, X):\n",
    "        for blk in self.net:\n",
    "            Y = blk(X)\n",
    "            # Concatenate input and output of each block along the channels\n",
    "            X = torch.cat((X, Y), dim=1)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "babe8f3c",
   "metadata": {},
   "source": [
    "### 8.7.3 Transition Layers\n",
    "\n",
    "Each denseblock increases the number of channels. Transition layers seek to regularize the numebr of channels to prevent excess complexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25870669",
   "metadata": {},
   "outputs": [],
   "source": [
    "def transition_block(num_channels):\n",
    "    return nn.Sequential(\n",
    "        nn.LazyBatchNorm2d(), nn.ReLU(),\n",
    "        nn.LazyConv2d(num_channels, kernel_size=1),\n",
    "        nn.AvgPool2d(kernel_size=2, stride=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4210dbc1",
   "metadata": {},
   "source": [
    "### DenseNet Model\n",
    "\n",
    "The stem is more or less the same as ResNet -> GoogLeNet => AlexNet but the body as the alternatign dense and transition blocks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c21763ad",
   "metadata": {},
   "source": [
    "## 8.8 Designing Convolution Network Architectures\n",
    "\n",
    "Neural architecture searches (NAS) are usually quite costly but are worth exploring."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0a917e",
   "metadata": {},
   "source": [
    "### 8.8.1 AnyNet Design Space\n",
    "\n",
    "AnyNet consists of stem, body, and head. \n",
    "\n",
    "* Stem:\n",
    "    * Take input images and use 3x3 Conv\n",
    "    * BN halves resolution donw to 1/2\n",
    "    * Generate c0 channels as input for body\n",
    "* Body:\n",
    "    * Can have a depth of d_i per stage\n",
    "    * Output channels at each layer of c_i\n",
    "    * Number of groups g_i\n",
    "    * Bottleneck ratios k_i\n",
    "* Head:\n",
    "    * Pooling operations\n",
    "    * Convolution to get to specific channels\n",
    "    * Pass to dense layers\n",
    "    * Dense layers cut down to n_classes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f48aa1",
   "metadata": {},
   "source": [
    "### 8.8.2 Distributions and Parameters of Design Spaces\n",
    "\n",
    "Params of the design space are hyperparams of the network instance. However due to permutations of possibilities exploding with network depth its infeasible to brute force in most cases. Instead finding strategies to determine better guidelines may have a higher yeild in a constrained env.\n",
    "\n",
    "1. Assume general design principles actually exist and compliant networks offer good performance. Assumes distribution over networks\n",
    "2. Need not train networks to convergence to assess performance but instead the intermediate results will be enough. Multi-fidelity optimization requires only a few passes through the data.\n",
    "3. Results on small scale generalize to larger scale, optimize over toy models and verify at scale\n",
    "4. Aspects of the design can be factorized independently of each other\n",
    "\n",
    "**How well do these assumptions hold**\n",
    "\n",
    "$$ F(e,p) = P_{net~p}{e(net) \\leq e} $$\n",
    "\n",
    "Find the distribution p over networks s.t. the nets have an error less than or equal to the other distributions p'. Take a sample of networks with errors from p and use the empirical CDF to find:\n",
    "\n",
    "$$ \\hat{F}(e,Z) = \\frac{1}{n} \\sum_{i=1}^n(e_i \\leq e)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c411ca6",
   "metadata": {},
   "source": [
    "### 8.8.3 RegNet\n",
    "\n",
    "Architecture that came out of the AnyNet design space\n",
    "\n",
    "* shares bottleneck ratios across design stages\n",
    "* shares group widths \n",
    "* increases channels across stages\n",
    "* increases network depths across stages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d0f8717",
   "metadata": {},
   "source": [
    "### 8.8.5 Discussion\n",
    "\n",
    "Transformers have a significant degree less inductive biases than CNNs for images due to the insane amount of training from images. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}