LINUX_TESTING.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RiAMJx7tr5-I"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\"> Import the Libraries</font>**\n",
        "\n",
        "We will start by installing and importing the required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X3AdbpZFCRR0"
      },
      "outputs": [],
      "source": [
        "# Import the required libraries.\n",
        "import os\n",
        "import cv2\n",
        "#import pafy\n",
        "import math\n",
        "import random\n",
        "import numpy as np\n",
        "import datetime as dt\n",
        "import tensorflow as tf\n",
        "from collections import deque\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "#from moviepy.editor import *\n",
        "%matplotlib inline\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "from tensorflow.keras.layers import *\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from tensorflow.keras.utils import plot_model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "nxRZOCSfJkpc"
      },
      "source": [
        "And will set `Numpy`, `Python`, and `Tensorflow` seeds to get consistent results on every execution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9AFIAq0Q0VwG"
      },
      "outputs": [],
      "source": [
        "seed_constant = 27\n",
        "np.random.seed(seed_constant)\n",
        "random.seed(seed_constant)\n",
        "tf.random.set_seed(seed_constant)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EfC0aLb3byRX"
      },
      "outputs": [],
      "source": [
        "import pathlib\n",
        "data_dir = \"\""
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Mxsl74v8Ud2i"
      },
      "source": [
        "## **<font style=\"color:rgb(134,19,348)\">Step 2: Preprocess the Dataset</font>**\n",
        "\n",
        "Next, we will perform some preprocessing on the dataset. First, we will read the video files from the dataset and resize the frames of the videos to a fixed width and height, to reduce the computations and normalized the data to range `[0-1]` by dividing the pixel values with `255`, which makes convergence faster while training the network.\n",
        "\n",
        "*But first, let's initialize some constants.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gb4Fv7Ag-kwb"
      },
      "outputs": [],
      "source": [
        "# Specify the height and width to which each video frame will be resized in our dataset.\n",
        "IMAGE_HEIGHT , IMAGE_WIDTH = 64, 64\n",
        "\n",
        "# Specify the number of frames of a video that will be fed to the model as one sequence.\n",
        "SEQUENCE_LENGTH = 10\n",
        "\n",
        "# Specify the directory containing the UCF50 dataset. \n",
        "DATASET_DIR = \"\"\n",
        "\n",
        "# Specify the list containing the names of the classes used for training. Feel free to choose any set of classes.\n",
        "CLASSES_LIST = []\n",
        "print(CLASSES_LIST)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_v6w2EzbvhRg"
      },
      "source": [
        "**Note:** *The **`IMAGE_HEIGHT`**, **`IMAGE_WIDTH`** and **`SEQUENCE_LENGTH`** constants can be increased for better results, although increasing the sequence length is only effective to a certain point, and increasing the values will result in the process being more computationally expensive.*\n",
        "\n",
        "### **<font style=\"color:rgb(134,19,348)\">Create a Function to Extract, Resize & Normalize Frames</font>**\n",
        "\n",
        "We will create a function **`frames_extraction()`** that will create a list containing the resized and normalized frames of a video whose path is passed to it as an argument. The function will read the video file frame by frame, although not all frames are added to the list as we will only need an evenly distributed sequence length of frames. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zBu224OG-szz"
      },
      "outputs": [],
      "source": [
        "def frames_extraction(video_path):\n",
        "    '''\n",
        "    This function will extract the required frames from a video after resizing and normalizing them.\n",
        "    Args:\n",
        "        video_path: The path of the video in the disk, whose frames are to be extracted.\n",
        "    Returns:\n",
        "        frames_list: A list containing the resized and normalized frames of the video.\n",
        "    '''\n",
        "\n",
        "    # Declare a list to store video frames.\n",
        "    frames_list = []\n",
        "    \n",
        "    # Read the Video File using the VideoCapture object.\n",
        "    video_reader = cv2.VideoCapture(video_path)\n",
        "\n",
        "    # Get the total number of frames in the video.\n",
        "    video_frames_count = int(video_reader.get(cv2.CAP_PROP_FRAME_COUNT))\n",
        "\n",
        "    # Calculate the the interval after which frames will be added to the list.\n",
        "    skip_frames_window = max(int(video_frames_count/SEQUENCE_LENGTH), 1)\n",
        "\n",
        "    # Iterate through the Video Frames.\n",
        "    for frame_counter in range(SEQUENCE_LENGTH):\n",
        "\n",
        "        # Set the current frame position of the video.\n",
        "        video_reader.set(cv2.CAP_PROP_POS_FRAMES, frame_counter * skip_frames_window)\n",
        "\n",
        "        # Reading the frame from the video. \n",
        "        success, frame = video_reader.read() \n",
        "\n",
        "        # Check if Video frame is not successfully read then break the loop\n",
        "        if not success:\n",
        "            break\n",
        "\n",
        "        # Resize the Frame to fixed height and width.\n",
        "        resized_frame = cv2.resize(frame, (IMAGE_HEIGHT, IMAGE_WIDTH))\n",
        "        \n",
        "        # Normalize the resized frame by dividing it with 255 so that each pixel value then lies between 0 and 1\n",
        "        normalized_frame = resized_frame / 255\n",
        "        \n",
        "        # Append the normalized frame into the frames list\n",
        "        frames_list.append(normalized_frame)\n",
        "    \n",
        "    # Release the VideoCapture object. \n",
        "    video_reader.release()\n",
        "\n",
        "    # Return the frames list.\n",
        "    #print(frames_list)\n",
        "    return frames_list"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "fL6Y0VupxLom"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Create a Function for Dataset Creation</font>**\n",
        "\n",
        "Now we will create a function **`create_dataset()`** that will iterate through all the classes specified in the **`CLASSES_LIST`** constant and will call the function  **`frame_extraction()`** on every video file of the selected classes and return the frames (**`features`**), class index ( **`labels`**), and video file path (**`video_files_paths`**)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j6MLp9PpDJ4-"
      },
      "outputs": [],
      "source": [
        "def create_dataset():\n",
        "    '''\n",
        "    This function will extract the data of the selected classes and create the required dataset.\n",
        "    Returns:\n",
        "        features:          A list containing the extracted frames of the videos.\n",
        "        labels:            A list containing the indexes of the classes associated with the videos.\n",
        "        video_files_paths: A list containing the paths of the videos in the disk.\n",
        "    '''\n",
        "\n",
        "    # Declared Empty Lists to store the features, labels and video file path values.\n",
        "    features = []\n",
        "    labels = []\n",
        "    video_files_paths = []\n",
        "    \n",
        "    # Iterating through all the classes mentioned in the classes list\n",
        "    for class_index, class_name in enumerate(CLASSES_LIST):\n",
        "        \n",
        "        # Display the name of the class whose data is being extracted.\n",
        "        print(f'Extracting Data of Class: {class_name}')\n",
        "        \n",
        "        # Get the list of video files present in the specific class name directory.\n",
        "        files_list = os.listdir(os.path.join(DATASET_DIR, class_name))\n",
        "        print(files_list)\n",
        "        \n",
        "        # Iterate through all the files present in the files list.\n",
        "        for file_name in files_list:\n",
        "                #print(len(file_name))\n",
        "                #print(vn)\n",
        "              \n",
        "                # Get the complete video path.\n",
        "                #print(sn)\n",
        "                video_file_path = os.path.join(DATASET_DIR, class_name, file_name)\n",
        "\n",
        "                # Extract the frames of the video file.\n",
        "                frames = frames_extraction(video_file_path)\n",
        "                #print(video_file_path)\n",
        "\n",
        "                # Check if the extracted frames are equal to the SEQUENCE_LENGTH specified above.\n",
        "                # So ignore the vides having frames less than the SEQUENCE_LENGTH.\n",
        "                if len(frames) == SEQUENCE_LENGTH:\n",
        "\n",
        "                    # Append the data to their repective lists.\n",
        "                    features.append(frames)\n",
        "                    labels.append(class_index)\n",
        "                    video_files_paths.append(video_file_path)\n",
        "\n",
        "    # Converting the list to numpy arrays\n",
        "    features = np.asarray(features)\n",
        "    labels = np.array(labels)  \n",
        "    \n",
        "    # Return the frames, class index, and video file path.\n",
        "    #print(video_files_paths)\n",
        "    return features, labels, video_files_paths"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "3jNu1QTX0G0q"
      },
      "source": [
        "Now we will utilize the function **`create_dataset()`** created above to extract the data of the selected classes and create the required dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ckAYgSriJN_R"
      },
      "outputs": [],
      "source": [
        "# Create the dataset.\n",
        "features, labels, video_files_paths = create_dataset()\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "goRGrc8_0Usb"
      },
      "source": [
        "Now we will convert `labels` (class indexes) into one-hot encoded vectors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KfcpuV6_Bd_T"
      },
      "outputs": [],
      "source": [
        "# Using Keras's to_categorical method to convert labels into one-hot-encoded vectors\n",
        "\n",
        "one_hot_encoded_labels = to_categorical(labels)\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PzRt7pP-0j5V"
      },
      "source": [
        "## **<font style=\"color:rgb(134,19,348)\">Step 3: Split the Data into Train and Test Set</font>**\n",
        "\n",
        "As of now, we have the required **`features`** (a NumPy array containing all the extracted frames of the videos) and **`one_hot_encoded_labels`** (also a Numpy array containing all class labels in one hot encoded format). So now, we will split our data to create training and testing sets. We will also shuffle the dataset before the split to avoid any bias and get splits representing the overall distribution of the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UhkQjq1JJSO2"
      },
      "outputs": [],
      "source": [
        "# Split the Data into Train ( 75% ) and Test Set ( 25% ).\n",
        "features_train, features_test, labels_train, labels_test = train_test_split(features, one_hot_encoded_labels,\n",
        "                                                                            test_size = 0.25, shuffle = True,\n",
        "                                                                            random_state = seed_constant)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "x8Ui5hLP1P9Q"
      },
      "source": [
        "## **<font style=\"color:rgb(134,19,348)\">Step 4: Implement the ConvLSTM Approach</font>**\n",
        "\n",
        "In this step, we will implement the first approach by using a combination of ConvLSTM cells. A ConvLSTM cell is a variant of an LSTM network that contains convolutions operations in the network. it is an LSTM with convolution embedded in the architecture, which makes it capable of identifying spatial features of the data while keeping into account the temporal relation. \n",
        "\n",
        "<center>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1KHN_JFWJoJi1xQj_bRdxy2QgevGOH1qP\" width= 500px>\n",
        "</center>\n",
        "\n",
        "\n",
        "For video classification, this approach effectively captures the spatial relation in the individual frames and the temporal relation across the different frames. As a result of this convolution structure, the ConvLSTM is capable of taking in 3-dimensional input `(width, height, num_of_channels)` whereas a simple LSTM only takes in 1-dimensional input hence an LSTM is incompatible for modeling Spatio-temporal data on its own.\n",
        "\n",
        "You can read the paper [**Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting**](https://arxiv.org/abs/1506.04214v1) by **Xingjian Shi** (NIPS 2015), to learn more about this architecture.\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Zx7PoAS7iEHM"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 4.1: Construct the Model</font>**\n",
        "\n",
        "\n",
        "To construct the model, we will use Keras [**`ConvLSTM2D`**](https://keras.io/api/layers/recurrent_layers/conv_lstm2d) recurrent layers. The **`ConvLSTM2D`** layer also takes in the number of filters and kernel size required for applying the convolutional operations. The output of the layers is flattened in the end and is fed to the **`Dense`** layer with softmax activation which outputs the probability of each action category. \n",
        "\n",
        "We will also use **`MaxPooling3D`** layers to reduce the dimensions of the frames and avoid unnecessary computations and **`Dropout`** layers to prevent [overfitting](https://en.wikipedia.org/wiki/Overfitting) the model on the data. The architecture is a simple one and has a small number of trainable parameters. This is because we are only dealing with a small subset of the dataset which does not require a large-scale model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h8cDSpfXJXwx"
      },
      "outputs": [],
      "source": [
        "def create_convlstm_model():\n",
        "    '''\n",
        "    This function will construct the required convlstm model.\n",
        "    Returns:\n",
        "        model: It is the required constructed convlstm model.\n",
        "    '''\n",
        "\n",
        "    # We will use a Sequential model for model construction\n",
        "    model = Sequential()\n",
        "\n",
        "    # Define the Model Architecture.\n",
        "    ########################################################################################################################\n",
        "    \n",
        "    model.add(ConvLSTM2D(filters = 4, kernel_size = (3, 3), activation = 'tanh',data_format = \"channels_last\",\n",
        "                         recurrent_dropout=0.2, return_sequences=True, input_shape = (SEQUENCE_LENGTH,\n",
        "                                                                                      IMAGE_HEIGHT, IMAGE_WIDTH, 3)))\n",
        "    \n",
        "    model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))\n",
        "    model.add(TimeDistributed(Dropout(0.2)))\n",
        "    \n",
        "    model.add(ConvLSTM2D(filters = 8, kernel_size = (3, 3), activation = 'tanh', data_format = \"channels_last\",\n",
        "                         recurrent_dropout=0.2, return_sequences=True))\n",
        "    \n",
        "    model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))\n",
        "    model.add(TimeDistributed(Dropout(0.2)))\n",
        "    \n",
        "    model.add(ConvLSTM2D(filters = 14, kernel_size = (3, 3), activation = 'tanh', data_format = \"channels_last\",\n",
        "                         recurrent_dropout=0.2, return_sequences=True))\n",
        "    \n",
        "    model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))\n",
        "    model.add(TimeDistributed(Dropout(0.2)))\n",
        "    \n",
        "    model.add(ConvLSTM2D(filters = 16, kernel_size = (3, 3), activation = 'tanh', data_format = \"channels_last\",\n",
        "                         recurrent_dropout=0.2, return_sequences=True))\n",
        "    \n",
        "    model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))\n",
        "    #model.add(TimeDistributed(Dropout(0.2)))\n",
        "    \n",
        "    model.add(Flatten()) \n",
        "    \n",
        "    model.add(Dense(len(CLASSES_LIST), activation = \"softmax\"))\n",
        "    \n",
        "    ########################################################################################################################\n",
        "     \n",
        "    # Display the models summary.\n",
        "    model.summary()\n",
        "    \n",
        "    # Return the constructed convlstm model.\n",
        "    return model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "KL9du-YJJkp5"
      },
      "source": [
        "Now we will utilize the function **`create_convlstm_model()`** created above, to construct the required `convlstm` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2D1CS2Z6Jkp5"
      },
      "outputs": [],
      "source": [
        "# Construct the required convlstm model.\n",
        "convlstm_model = create_convlstm_model()\n",
        "\n",
        "# Display the success message. \n",
        "print(\"Model Created Successfully!\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "OOGUT5DbK7Sr"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Check Model’s Structure:</font>**\n",
        "\n",
        "\n",
        "Now we will use the **`plot_model()`** function, to check the structure of the constructed model, this is helpful while constructing a complex network and making that the network is created correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hbSK_rKUK-xQ"
      },
      "outputs": [],
      "source": [
        "# Plot the structure of the contructed model.\n",
        "plot_model(convlstm_model, to_file = 'convlstm_model_structure_plot.png', show_shapes = True, show_layer_names = True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHKkZvkCOnMU"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 4.2: Compile & Train the Model</font>**\n",
        "\n",
        "Next, we will add an early stopping callback to prevent [overfitting](https://en.wikipedia.org/wiki/Overfitting) and start the training after compiling the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Yxx_adaPr8Vf"
      },
      "outputs": [],
      "source": [
        "# Create an Instance of Early Stopping Callback\n",
        "early_stopping_callback = EarlyStopping(monitor = 'val_loss', patience = 10, mode = 'min', restore_best_weights = True)\n",
        "\n",
        "# Compile the model and specify loss function, optimizer and metrics values to the model\n",
        "convlstm_model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam', metrics = [\"accuracy\"])\n",
        "\n",
        "# Start training the model.\n",
        "convlstm_model_training_history = convlstm_model.fit(x = features_train, y = labels_train, epochs = 100, batch_size = 4,\n",
        "                                                     shuffle = True, validation_split = 0.2, \n",
        "                                                     callbacks = [early_stopping_callback])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "z1KR_kfz2oFl"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Evaluate the Trained Model</font>**\n",
        "\n",
        "After training, we will evaluate the model on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "74tGjokkmSHR"
      },
      "outputs": [],
      "source": [
        "# Evaluate the trained model.\n",
        "model_evaluation_history = convlstm_model.evaluate(features_test, labels_test)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "TGglb-k867iK"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Save the Model</font>**\n",
        "\n",
        "Now we will save the model to avoid training it from scratch every time we need the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GLfnP_czYudN"
      },
      "outputs": [],
      "source": [
        "# Get the loss and accuracy from model_evaluation_history.\n",
        "model_evaluation_loss, model_evaluation_accuracy = model_evaluation_history\n",
        "\n",
        "# Define the string date format.\n",
        "# Get the current Date and Time in a DateTime Object.\n",
        "# Convert the DateTime object to string according to the style mentioned in date_time_format string.\n",
        "date_time_format = '%Y_%m_%d__%H_%M_%S'\n",
        "current_date_time_dt = dt.datetime.now()\n",
        "current_date_time_string = dt.datetime.strftime(current_date_time_dt, date_time_format)\n",
        "\n",
        "# Define a useful name for our model to make it easy for us while navigating through multiple saved models.\n",
        "model_file_name = f'convlstm_model___Date_Time_{current_date_time_string}___Loss_{model_evaluation_loss}___Accuracy_{model_evaluation_accuracy}.h5'\n",
        "\n",
        "# Save your Model.\n",
        "convlstm_model.save(model_file_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "BSeAfu6uZdai"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 4.3: Plot Model’s Loss & Accuracy Curves</font>**\n",
        "\n",
        "\n",
        "Now we will create a function **`plot_metric()`** to visualize the training and validation metrics. We already have separate metrics from our training and validation steps so now we just have to visualize them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qrQnmVF7aZjQ"
      },
      "outputs": [],
      "source": [
        "def plot_metric(model_training_history, metric_name_1, metric_name_2, plot_name):\n",
        "    '''\n",
        "    This function will plot the metrics passed to it in a graph.\n",
        "    Args:\n",
        "        model_training_history: A history object containing a record of training and validation \n",
        "                                loss values and metrics values at successive epochs\n",
        "        metric_name_1:          The name of the first metric that needs to be plotted in the graph.\n",
        "        metric_name_2:          The name of the second metric that needs to be plotted in the graph.\n",
        "        plot_name:              The title of the graph.\n",
        "    '''\n",
        "    \n",
        "    # Get metric values using metric names as identifiers.\n",
        "    metric_value_1 = model_training_history.history[metric_name_1]\n",
        "    metric_value_2 = model_training_history.history[metric_name_2]\n",
        "    \n",
        "    # Construct a range object which will be used as x-axis (horizontal plane) of the graph.\n",
        "    epochs = range(len(metric_value_1))\n",
        "\n",
        "    # Plot the Graph.\n",
        "    plt.plot(epochs, metric_value_1, 'blue', label = metric_name_1)\n",
        "    plt.plot(epochs, metric_value_2, 'red', label = metric_name_2)\n",
        "\n",
        "    # Add title to the plot.\n",
        "    plt.title(str(plot_name))\n",
        "\n",
        "    # Add legend to the plot.\n",
        "    plt.legend()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tZzhYAQbJkp9"
      },
      "source": [
        "Now we will utilize the function **`plot_metric()`** created above, to visualize and understand the metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tKABbscJblP_"
      },
      "outputs": [],
      "source": [
        "# Visualize the training and validation loss metrices.\n",
        "plot_metric(convlstm_model_training_history, 'loss', 'val_loss', 'Stealing Class Total Loss vs Total Validation Loss')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cTSpJP-bb7WV"
      },
      "outputs": [],
      "source": [
        "# Visualize the training and validation accuracy metrices.\n",
        "plot_metric(convlstm_model_training_history, 'accuracy', 'val_accuracy', 'Stealing Class Total Accuracy vs Total Validation Accuracy') "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "clp7BFVDBNxp"
      },
      "source": [
        "## **<font style=\"color:rgb(134,19,348)\">Step 5: Implement the LRCN Approach</font>**\n",
        "\n",
        "In this step, we will implement the LRCN Approach by combining Convolution and LSTM layers in a single model. Another similar approach can be to use a CNN model and LSTM model trained separately. The CNN model can be used to extract spatial features from the frames in the video, and for this purpose, a pre-trained model can be used, that can be fine-tuned for the problem. And the LSTM model can then use the features extracted by CNN, to predict the action being performed in the video. \n",
        "\n",
        "\n",
        "But here, we will implement another approach known as the Long-term Recurrent Convolutional Network (LRCN), which combines CNN and LSTM layers in a single model. The Convolutional layers are used for spatial feature extraction from the frames, and the extracted spatial features are fed to LSTM layer(s) at each time-steps for temporal sequence modeling. This way the network learns spatiotemporal features directly in an end-to-end training, resulting in a robust model.\n",
        "\n",
        "<center>\n",
        "<img src='https://drive.google.com/uc?export=download&id=1I-q5yLsIoNh2chfzT7JYvra17FsXvdme'>\n",
        "</center>\n",
        "\n",
        "\n",
        "You can read the paper [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://arxiv.org/abs/1411.4389?source=post_page---------------------------) by Jeff Donahue (CVPR 2015), to learn more about this architecture.\n",
        "\n",
        "We will also use [**`TimeDistributed`**](https://keras.io/api/layers/recurrent_layers/time_distributed/) wrapper layer, which allows applying the same layer to every frame of the video independently. So it makes a layer (around which it is wrapped) capable of taking input of shape `(no_of_frames, width, height, num_of_channels)` if originally the layer's input shape was `(width, height, num_of_channels)` which is very beneficial as it allows to input the whole video into the model in a single shot. \n",
        "\n",
        "<center>\n",
        "<img src='https://drive.google.com/uc?export=download&id=1CbauSm5XTY7ypHYBHH7rDSnJ5LO9CUWX' width=400>\n",
        "</center>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "pyKaiYrQ4iM6"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 5.1: Construct the Model</font>**\n",
        "\n",
        "To implement our LRCN architecture, we will use time-distributed **`Conv2D`** layers which will be followed by **`MaxPooling2D`** and **`Dropout`** layers. The feature extracted from the **`Conv2D`** layers will be then flattened using the  **`Flatten`** layer and will be fed to a **`LSTM`** layer. The **`Dense`** layer with softmax activation will then use the output from the **`LSTM`** layer to predict the action being performed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HKM766mB1fIv"
      },
      "outputs": [],
      "source": [
        "def create_LRCN_model():\n",
        "    '''\n",
        "    This function will construct the required LRCN model.\n",
        "    Returns:\n",
        "        model: It is the required constructed LRCN model.\n",
        "    '''\n",
        "\n",
        "    # We will use a Sequential model for model construction.\n",
        "    model = Sequential()\n",
        "    \n",
        "    # Define the Model Architecture.\n",
        "    ########################################################################################################################\n",
        "    \n",
        "    model.add(TimeDistributed(Conv2D(16, (3, 3), padding='same',activation = 'relu'),\n",
        "                              input_shape = (SEQUENCE_LENGTH, IMAGE_HEIGHT, IMAGE_WIDTH, 3)))\n",
        "    \n",
        "    model.add(TimeDistributed(MaxPooling2D((4, 4)))) \n",
        "    model.add(TimeDistributed(Dropout(0.25)))\n",
        "    \n",
        "    model.add(TimeDistributed(Conv2D(32, (3, 3), padding='same',activation = 'relu')))\n",
        "    model.add(TimeDistributed(MaxPooling2D((4, 4))))\n",
        "    model.add(TimeDistributed(Dropout(0.25)))\n",
        "    \n",
        "    model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))\n",
        "    model.add(TimeDistributed(MaxPooling2D((2, 2))))\n",
        "    model.add(TimeDistributed(Dropout(0.25)))\n",
        "    \n",
        "    model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))\n",
        "    model.add(TimeDistributed(MaxPooling2D((2, 2))))\n",
        "    #model.add(TimeDistributed(Dropout(0.25)))\n",
        "                                      \n",
        "    model.add(TimeDistributed(Flatten()))\n",
        "                                      \n",
        "    model.add(LSTM(32))\n",
        "                                      \n",
        "    model.add(Dense(len(CLASSES_LIST), activation = 'softmax'))\n",
        "\n",
        "    ########################################################################################################################\n",
        "\n",
        "    # Display the models summary.\n",
        "    model.summary()\n",
        "    \n",
        "    # Return the constructed LRCN model.\n",
        "    return model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bHUPdK-fJkqD"
      },
      "source": [
        "Now we will utilize the function **`create_LRCN_model()`** created above to construct the required `LRCN` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZJjC2VA2JkqD"
      },
      "outputs": [],
      "source": [
        "# Construct the required LRCN model.\n",
        "LRCN_model = create_LRCN_model()\n",
        "\n",
        "# Display the success message.\n",
        "print(\"Model Created Successfully!\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Px_gEcDbA5k4"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Check Model’s Structure:</font>**\n",
        "\n",
        "Now we will use the **`plot_model()`** function to check the structure of the constructed `LRCN` model. As we had checked for the previous model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X8nAG3xYA5lW"
      },
      "outputs": [],
      "source": [
        "# Plot the structure of the contructed LRCN model.\n",
        "plot_model(LRCN_model, to_file = 'LRCN_model_structure_plot.png', show_shapes = True, show_layer_names = True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "wsDrzxCDA5lX"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 5.2: Compile & Train the Model</font>**\n",
        "\n",
        "\n",
        "After checking the structure, we will compile and start training the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mtrQGXc8A5lY"
      },
      "outputs": [],
      "source": [
        "# Create an Instance of Early Stopping Callback.\n",
        "early_stopping_callback = EarlyStopping(monitor = 'val_loss', patience = 15, mode = 'min', restore_best_weights = True)\n",
        " \n",
        "# Compile the model and specify loss function, optimizer and metrics to the model.\n",
        "LRCN_model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam', metrics = [\"accuracy\"])\n",
        "\n",
        "# Start training the model.\n",
        "LRCN_model_training_history = LRCN_model.fit(x = features_train, y = labels_train, epochs = 100, batch_size = 4 ,\n",
        "                                             shuffle = True, validation_split = 0.2, callbacks = [early_stopping_callback])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tMepQus5A5la"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Evaluating the trained Model</font>**\n",
        "\n",
        "As done for the previous one, we will evaluate the `LRCN` model on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZwSYdhEFA5lb"
      },
      "outputs": [],
      "source": [
        "# Evaluate the trained model.\n",
        "model_evaluation_history = LRCN_model.evaluate(features_test, labels_test)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_SNLhkFNA5lb"
      },
      "source": [
        "#### **<font style=\"color:rgb(134,19,348)\">Save the Model</font>**\n",
        "\n",
        "After that, we will save the model for future uses using the same technique we had used for the previous model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R6nD8oMqA5lc"
      },
      "outputs": [],
      "source": [
        "# Get the loss and accuracy from model_evaluation_history.\n",
        "model_evaluation_loss, model_evaluation_accuracy = model_evaluation_history\n",
        "\n",
        "# Define the string date format.\n",
        "# Get the current Date and Time in a DateTime Object.\n",
        "# Convert the DateTime object to string according to the style mentioned in date_time_format string.\n",
        "date_time_format = '%Y_%m_%d__%H_%M_%S'\n",
        "current_date_time_dt = dt.datetime.now()\n",
        "current_date_time_string = dt.datetime.strftime(current_date_time_dt, date_time_format)\n",
        "    \n",
        "# Define a useful name for our model to make it easy for us while navigating through multiple saved models.\n",
        "model_file_name = f'LRCN_model___Date_Time_{current_date_time_string}___Loss_{model_evaluation_loss}___Accuracy_{model_evaluation_accuracy}.h5'\n",
        "\n",
        "# Save the Model.\n",
        "LRCN_model.save(model_file_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bDs61KA-A5lc"
      },
      "source": [
        "### **<font style=\"color:rgb(134,19,348)\">Step 5.3: Plot Model’s Loss & Accuracy Curves</font>**\n",
        "\n",
        "Now we will utilize the function **`plot_metric()`** we had created above to visualize the training and validation metrics of this model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nvAgcD_FA5ld"
      },
      "outputs": [],
      "source": [
        "# Visualize the training and validation loss metrices.\n",
        "plot_metric(LRCN_model_training_history, 'loss', 'val_loss', 'Total Loss vs Total Validation Loss')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YZHXzXJZA5le"
      },
      "outputs": [],
      "source": [
        "# Visualize the training and validation accuracy metrices.\n",
        "plot_metric(LRCN_model_training_history, 'accuracy', 'val_accuracy', 'Total Accuracy vs Total Validation Accuracy')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "private_outputs": true,
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}