04_RNN_LSTM/EVA_P2S3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "EVA P2S3.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jofyc9OC4Qcf"
      },
      "source": [
        "#Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ahBVnrNc3E0U"
      },
      "source": [
        "import numpy as np\n",
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from IPython import display\n",
        "plt.style.use('seaborn-white')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crQSAaIz4SkA"
      },
      "source": [
        "# Read and process data. \n",
        "\n",
        "Download the file from this URL: https://drive.google.com/file/d/1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS/view?usp=sharing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgOGxPDP3Wpp"
      },
      "source": [
        "data = open('text.txt', 'r').read()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeXXMLRb4kXb"
      },
      "source": [
        "Process data and calculate indices"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5TKeiOp4jtl"
      },
      "source": [
        "chars = list(set(data))\n",
        "data_size, X_size = len(data), len(chars)\n",
        "print(\"Corona Virus article has %d characters, %d unique characters\" %(data_size, X_size))\n",
        "char_to_idx = {ch:i for i,ch in enumerate(chars)}\n",
        "idx_to_char = {i:ch for i,ch in enumerate(chars)}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4C53MB135LRY"
      },
      "source": [
        "# Constants and Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dfj21ORa49Ps"
      },
      "source": [
        "Hidden_Layer_size = 10 #size of the hidden layer\n",
        "Time_steps = 10 # Number of time steps (length of the sequence) used for training\n",
        "learning_rate = 1e-1 # Learning Rate\n",
        "weight_sd = 0.1 #Standard deviation of weights for initialization\n",
        "z_size = Hidden_Layer_size + X_size #Size of concatenation(H, X) vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OdmJf4Du5uhb"
      },
      "source": [
        "# Activation Functions and Derivatives"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seGHei_D5FGk"
      },
      "source": [
        "def sigmoid(x): # sigmoid function\n",
        "  return # write your code here\n",
        "\n",
        "def dsigmoid(y): # derivative of sigmoid function\n",
        "  return # write your code here\n",
        "\n",
        "def tanh(x): # tanh function\n",
        "  return # write your code here\n",
        "\n",
        "def dtanh(y): # derivative of tanh\n",
        "  return # write your code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KeCvVH1v6Me-"
      },
      "source": [
        "# Quiz Question 1\n",
        "\n",
        "What is the value of sigmoid(0) calculated from  your code? (Answer up to 1 decimal point, e.g. 4.2 and NOT 4.29999999, no rounding off).\n",
        "\n",
        "# Quiz Question 2\n",
        "\n",
        "What is the value of dsigmoid(sigmoid(0)) calculated from your code?? (Answer up to 2 decimal point, e.g. 4.29 and NOT 4.29999999, no rounding off). \n",
        "\n",
        "# Quiz Question 3\n",
        "\n",
        "What is the value of tanh(dsigmoid(sigmoid(0))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off).\n",
        "\n",
        "# Quiz Question 4\n",
        "\n",
        "What is the value of dtanh(tanh(dsigmoid(sigmoid(0)))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeSVipDu8iKE"
      },
      "source": [
        "# Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICbWNemE6LGV"
      },
      "source": [
        "class Param:\n",
        "    def __init__(self, name, value):\n",
        "      self.name = name\n",
        "      self.v = value # parameter value\n",
        "      self.d = np.zeros_like(value) # derivative\n",
        "      self.m = np.zeros_like(value) # momentum for Adagrad"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j83pZNPE8212"
      },
      "source": [
        "We use random weights with normal distribution (0, weight_sd) for  tanh  activation function and (0.5, weight_sd) for  `sigmoid`  activation function.\n",
        "\n",
        "Biases are initialized to zeros."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "swHwLXOI9E7V"
      },
      "source": [
        "# LSTM \n",
        "You are making this network, please note f, i, c and o (also \"v\") in the image below:\n",
        "![alt text](http://blog.varunajayasiri.com/ml/lstm.svg)\n",
        "\n",
        "Please note that we are concatenating the old_hidden_vector and new_input."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0DBzNY-90s5"
      },
      "source": [
        "# Quiz Question 4\n",
        "\n",
        "In the class definition below, what should be size_a, size_b, and size_c? ONLY use the variables defined above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SFuHhqVq6Wge"
      },
      "source": [
        "size_a = # write your code here\n",
        "size_b = # write your code here\n",
        "size_c = # write your code here\n",
        "\n",
        "class Parameters:\n",
        "    def __init__(self):\n",
        "        self.W_f = Param('W_f', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_f = Param('b_f', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_i = Param('W_i', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_i = Param('b_i', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_C = Param('W_C', np.random.randn(size_a, size_b) * weight_sd)\n",
        "        self.b_C = Param('b_C', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_o = Param('W_o', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_o = Param('b_o', np.zeros((size_a, 1)))\n",
        "\n",
        "        #For final layer to predict the next character\n",
        "        self.W_v = Param('W_v', np.random.randn(X_size, size_a) * weight_sd)\n",
        "        self.b_v = Param('b_v', np.zeros((size_c, 1)))\n",
        "        \n",
        "    def all(self):\n",
        "        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,\n",
        "               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]\n",
        "        \n",
        "parameters = Parameters()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzmfGLZt_xVs"
      },
      "source": [
        "Look at these operations which we'll be writing:\n",
        "\n",
        "**Concatenation of h and x:**\n",
        "\n",
        "$z\\:=\\:\\left[h_{t-1},\\:x\\right]$\n",
        "\n",
        "$f_t=\\sigma\\left(W_f\\cdot z\\:+\\:b_f\\:\\right)$\n",
        "\n",
        "$i_i=\\sigma\\left(W_i\\cdot z\\:+\\:b_i\\right)$\n",
        "\n",
        "$\\overline{C_t}=\\tanh\\left(W_C\\cdot z\\:+\\:b_C\\right)$\n",
        "\n",
        "$C_t=f_t\\ast C_{t-1}+i_t\\ast \\overline{C}_t$\n",
        "\n",
        "$o_t=\\sigma\\left(W_o\\cdot z\\:+\\:b_i\\right)$\n",
        "\n",
        "$h_t=o_t\\ast\\tanh\\left(C_t\\right)$\n",
        "\n",
        "**Logits:**\n",
        "\n",
        "$v_t=W_v\\cdot h_t+b_v$\n",
        "\n",
        "**Softmax:**\n",
        "\n",
        "$\\hat{y}=softmax\\left(v_t\\right)$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-bUkseNnDott"
      },
      "source": [
        "def forward(x, h_prev, C_prev, p = parameters):\n",
        "    assert x.shape == (X_size, 1)\n",
        "    assert h_prev.shape == (Hidden_Layer_size, 1)\n",
        "    assert C_prev.shape == (Hidden_Layer_size, 1)\n",
        "    \n",
        "    z = np.row_stack((h_prev, x))\n",
        "    f = # write your code here\n",
        "    i = # write your code here\n",
        "    C_bar = # write your code here\n",
        "\n",
        "    C = # write your code here\n",
        "    o = # write your code here\n",
        "    h = # write your code here\n",
        "\n",
        "    v = # write your code here\n",
        "    y = np.exp(v) / np.sum(np.exp(v)) #softmax\n",
        "\n",
        "    return z, f, i, C_bar, C, o, h, v, y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZrDhZIjFpdI"
      },
      "source": [
        "You must finish the function above before you can attempt the questions below. \n",
        "\n",
        "# Quiz Question 5\n",
        "\n",
        "What is the output of 'print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))'?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XV-YVl_GGiX8"
      },
      "source": [
        "# Quiz Question 6. \n",
        "\n",
        "Assuming you have fixed the forward function, run this command: \n",
        "z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))\n",
        "\n",
        "Now, find these values:\n",
        "\n",
        "\n",
        "1.   print(z.shape)\n",
        "2.   print(np.sum(z))\n",
        "3.   print(np.sum(f))\n",
        "\n",
        "Copy and paste exact values you get in the logs into the quiz.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1GvKVWmTDt3H"
      },
      "source": [
        "z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NeSvhkqwILsG"
      },
      "source": [
        "# Backpropagation\n",
        "\n",
        "Here we are defining the backpropagation. It's too complicated, here is the whole code. (Please note that this would work only if your earlier code is perfect)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zIa1jUZiGPmF"
      },
      "source": [
        "def backward(target, dh_next, dC_next, C_prev,\n",
        "             z, f, i, C_bar, C, o, h, v, y,\n",
        "             p = parameters):\n",
        "    \n",
        "    assert z.shape == (X_size + Hidden_Layer_size, 1)\n",
        "    assert v.shape == (X_size, 1)\n",
        "    assert y.shape == (X_size, 1)\n",
        "    \n",
        "    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:\n",
        "        assert param.shape == (Hidden_Layer_size, 1)\n",
        "        \n",
        "    dv = np.copy(y)\n",
        "    dv[target] -= 1\n",
        "\n",
        "    p.W_v.d += np.dot(dv, h.T)\n",
        "    p.b_v.d += dv\n",
        "\n",
        "    dh = np.dot(p.W_v.v.T, dv)        \n",
        "    dh += dh_next\n",
        "    do = dh * tanh(C)\n",
        "    do = dsigmoid(o) * do\n",
        "    p.W_o.d += np.dot(do, z.T)\n",
        "    p.b_o.d += do\n",
        "\n",
        "    dC = np.copy(dC_next)\n",
        "    dC += dh * o * dtanh(tanh(C))\n",
        "    dC_bar = dC * i\n",
        "    dC_bar = dtanh(C_bar) * dC_bar\n",
        "    p.W_C.d += np.dot(dC_bar, z.T)\n",
        "    p.b_C.d += dC_bar\n",
        "\n",
        "    di = dC * C_bar\n",
        "    di = dsigmoid(i) * di\n",
        "    p.W_i.d += np.dot(di, z.T)\n",
        "    p.b_i.d += di\n",
        "\n",
        "    df = dC * C_prev\n",
        "    df = dsigmoid(f) * df\n",
        "    p.W_f.d += np.dot(df, z.T)\n",
        "    p.b_f.d += df\n",
        "\n",
        "    dz = (np.dot(p.W_f.v.T, df)\n",
        "         + np.dot(p.W_i.v.T, di)\n",
        "         + np.dot(p.W_C.v.T, dC_bar)\n",
        "         + np.dot(p.W_o.v.T, do))\n",
        "    dh_prev = dz[:Hidden_Layer_size, :]\n",
        "    dC_prev = f * dC\n",
        "    \n",
        "    return dh_prev, dC_prev"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tnc7WpRkIU5S"
      },
      "source": [
        "# Forward and Backward Combined Pass\n",
        "\n",
        "Let's first clear the gradients before each backward pass"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJWoC3U1ITf8"
      },
      "source": [
        "def clear_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.d.fill(0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7XN93UnjIgmA"
      },
      "source": [
        "Clip gradients to mitigate exploding gradients"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0LTsublxIfFl"
      },
      "source": [
        "def clip_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        np.clip(p.d, -1, 1, out=p.d)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7XUpDTWIl_Y"
      },
      "source": [
        "Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.\n",
        "\n",
        "input, target are list of integers, with character indexes.\n",
        "h_prev is the array of initial h at  h−1  (size H x 1)\n",
        "C_prev is the array of initial C at  C−1  (size H x 1)\n",
        "Returns loss, final  hT  and  CT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CQNxjTuZIia_"
      },
      "source": [
        "def forward_backward(inputs, targets, h_prev, C_prev):\n",
        "    global paramters\n",
        "    \n",
        "    # To store the values for each time step\n",
        "    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}\n",
        "    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}\n",
        "    v_s, y_s =  {}, {}\n",
        "    \n",
        "    # Values at t - 1\n",
        "    h_s[-1] = np.copy(h_prev)\n",
        "    C_s[-1] = np.copy(C_prev)\n",
        "    \n",
        "    loss = 0\n",
        "    # Loop through time steps\n",
        "    assert len(inputs) == Time_steps\n",
        "    for t in range(len(inputs)):\n",
        "        x_s[t] = np.zeros((X_size, 1))\n",
        "        x_s[t][inputs[t]] = 1 # Input character\n",
        "        \n",
        "        (z_s[t], f_s[t], i_s[t],\n",
        "        C_bar_s[t], C_s[t], o_s[t], h_s[t],\n",
        "        v_s[t], y_s[t]) = \\\n",
        "            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass\n",
        "            \n",
        "        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t\n",
        "        \n",
        "    clear_gradients()\n",
        "\n",
        "    dh_next = np.zeros_like(h_s[0]) #dh from the next character\n",
        "    dC_next = np.zeros_like(C_s[0]) #dh from the next character\n",
        "\n",
        "    for t in reversed(range(len(inputs))):\n",
        "        # Backward pass\n",
        "        dh_next, dC_next = \\\n",
        "            backward(target = targets[t], dh_next = dh_next,\n",
        "                     dC_next = dC_next, C_prev = C_s[t-1],\n",
        "                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],\n",
        "                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],\n",
        "                     y = y_s[t])\n",
        "\n",
        "    clip_gradients()\n",
        "        \n",
        "    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcy5u_vRItkV"
      },
      "source": [
        "# Sample the next character"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p8SrtJiwIsSm"
      },
      "source": [
        "def sample(h_prev, C_prev, first_char_idx, sentence_length):\n",
        "    x = np.zeros((X_size, 1))\n",
        "    x[first_char_idx] = 1\n",
        "\n",
        "    h = h_prev\n",
        "    C = C_prev\n",
        "\n",
        "    indexes = []\n",
        "    \n",
        "    for t in range(sentence_length):\n",
        "        _, _, _, _, C, _, h, _, p = forward(x, h, C)\n",
        "        idx = np.random.choice(range(X_size), p=p.ravel())\n",
        "        x = np.zeros((X_size, 1))\n",
        "        x[idx] = 1\n",
        "        indexes.append(idx)\n",
        "\n",
        "    return indexes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiWFaWLNIx_L"
      },
      "source": [
        "# Training (Adagrad)\n",
        "\n",
        "Update the graph and display a sample output\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ENQYU-7AIw0t"
      },
      "source": [
        "def update_status(inputs, h_prev, C_prev):\n",
        "    #initialized later\n",
        "    global plot_iter, plot_loss\n",
        "    global smooth_loss\n",
        "    \n",
        "    # Get predictions for 200 letters with current model\n",
        "\n",
        "    sample_idx = sample(h_prev, C_prev, inputs[0], 200)\n",
        "    txt = ''.join(idx_to_char[idx] for idx in sample_idx)\n",
        "\n",
        "    # Clear and plot\n",
        "    plt.plot(plot_iter, plot_loss)\n",
        "    display.clear_output(wait=True)\n",
        "    plt.show()\n",
        "\n",
        "    #Print prediction and loss\n",
        "    print(\"----\\n %s \\n----\" % (txt, ))\n",
        "    print(\"iter %d, loss %f\" % (iteration, smooth_loss))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACXcASJuI73a"
      },
      "source": [
        "# Update Parameters\n",
        "\n",
        "\\begin{align}\n",
        "\\theta_i &= \\theta_i - \\eta\\frac{d\\theta_i}{\\sum dw_{\\tau}^2} \\\\\n",
        "d\\theta_i &= \\frac{\\partial L}{\\partial \\theta_i}\n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bR08TvcjI4Pf"
      },
      "source": [
        "def update_paramters(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.m += p.d * p.d # Calculate sum of gradients\n",
        "        #print(learning_rate * dparam)\n",
        "        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "La9vyJ6RJLFK"
      },
      "source": [
        "To delay the keyboard interrupt to prevent the training from stopping in the middle of an iteration\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZVDHbMb7JNGT"
      },
      "source": [
        "# Exponential average of loss\n",
        "# Initialize to a error of a random model\n",
        "smooth_loss = -np.log(1.0 / X_size) * Time_steps\n",
        "\n",
        "iteration, pointer = 0, 0\n",
        "\n",
        "# For the graph\n",
        "plot_iter = np.zeros((0))\n",
        "plot_loss = np.zeros((0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HF6vS0VWJqsS"
      },
      "source": [
        "# Training Loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OQyNSL0iJOxH"
      },
      "source": [
        "iter = 1000\n",
        "while iter > 0:\n",
        "  # Reset\n",
        "  if pointer + Time_steps >= len(data) or iteration == 0:\n",
        "      g_h_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      g_C_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      pointer = 0\n",
        "\n",
        "\n",
        "  inputs = ([char_to_idx[ch] \n",
        "              for ch in data[pointer: pointer + Time_steps]])\n",
        "  targets = ([char_to_idx[ch] \n",
        "              for ch in data[pointer + 1: pointer + Time_steps + 1]])\n",
        "\n",
        "  loss, g_h_prev, g_C_prev = \\\n",
        "      forward_backward(inputs, targets, g_h_prev, g_C_prev)\n",
        "  smooth_loss = smooth_loss * 0.999 + loss * 0.001\n",
        "\n",
        "  # Print every hundred steps\n",
        "  if iteration % 100 == 0:\n",
        "      update_status(inputs, g_h_prev, g_C_prev)\n",
        "\n",
        "  update_paramters()\n",
        "\n",
        "  plot_iter = np.append(plot_iter, [iteration])\n",
        "  plot_loss = np.append(plot_loss, [loss])\n",
        "\n",
        "  pointer += Time_steps\n",
        "  iteration += 1\n",
        "  iter = iter -1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2AKpa1BGOItQ"
      },
      "source": [
        "# Quiz Question 7. \n",
        "\n",
        "Run the above code for 50000 iterations making sure that you have 100 hidden layers and time_steps is 40. What is the loss value you're seeing?"
      ]
    }
  ]
}