Basic_Temporal_Supervised_Machine_Learning_MLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/todnewman/coe_training/blob/master/Basic_Temporal_Supervised_Machine_Learning_MLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jO7hIpIYm72S"
      },
      "source": [
        "# Supervised Machine Learning for Time Series Data\n",
        "**Author**: W. Tod Newman\n",
        "\n",
        "**Updates**: New Version\n",
        "\n",
        "## Problem\n",
        "\n",
        "Determine the probability of rain some number of hours in the future based on the last X hours of collected data across six features.\n",
        "\n",
        "This notebook shows how to take advantage of the information contained in sequences from time series data.  It shows how to convert time series data into dataframes useful for supervised machine learning. \n",
        "\n",
        "Additionally, this notebook shows how to:\n",
        "1.  Follow a managed workflow from Data to Verification.  This uses GitHub, Google Drive, and Tensorboard to demonstrate what a production workflow might look like.\n",
        "2.  Connect a Keras/Tensorflow model with SKLearn and Pandas tools\n",
        "3.  Use Tensorboard for Evaluating models\n",
        "\n",
        "## Learning Objectives\n",
        "\n",
        "\n",
        "* Evaluate why time series data is interesting and useful for prediction.\n",
        "* Learn techniques for unrolling sequences and building a supervised learning dataset from time series data.\n",
        "* Learn techniques for shifting labels to enable predictions of future states using current data.\n",
        "* Demonstrate real-time inference using new data.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "This dataset is the famous \"Tod Newman Back Porch\" Dataset.  See https://todnewman.com/?p=66 for details on the data collection system.\n",
        "\n",
        "Features include:\n",
        "* Distance of Lightning Strike (km)\n",
        "* Energy of Lightning Strike (unitless)\n",
        "* Visible Light Luminosity (lux)\n",
        "* Humidity (%)\n",
        "* Pressure (HPa)\n",
        "* Temperature (deg F)\n",
        "\n",
        "The Label is:\n",
        "* Rain: Was it raining during the collection interval or not.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 745
        },
        "id": "skj9F394nV6k",
        "outputId": "a5d097ce-2fea-4947-d47b-a02845fa3783"
      },
      "source": [
        "import os, sys \n",
        "sys.path.insert(0, os.getcwd()) \n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import math\n",
        "import matplotlib\n",
        "from tensorflow import keras\n",
        "import sklearn as sk\n",
        "from sklearn import preprocessing\n",
        "from sklearn import metrics\n",
        "#from sklearn.grid_search import GridSearchCV, RandomizedSearchCV\n",
        "from sklearn import linear_model\n",
        "%matplotlib inline\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
        "tensorboard = False\n",
        "\n",
        "!pip install pycodestyle pycodestyle_magic\n",
        "!pip install flake8\n",
        "%load_ext pycodestyle_magic\n",
        "\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pycodestyle\n",
            "  Downloading pycodestyle-2.10.0-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[K     |████████████████████████████████| 41 kB 37 kB/s \n",
            "\u001b[?25hCollecting pycodestyle_magic\n",
            "  Downloading pycodestyle_magic-0.5-py2.py3-none-any.whl (9.5 kB)\n",
            "Installing collected packages: pycodestyle-magic, pycodestyle\n",
            "Successfully installed pycodestyle-2.10.0 pycodestyle-magic-0.5\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting flake8\n",
            "  Downloading flake8-5.0.4-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[K     |████████████████████████████████| 61 kB 75 kB/s \n",
            "\u001b[?25hCollecting pyflakes<2.6.0,>=2.5.0\n",
            "  Downloading pyflakes-2.5.0-py2.py3-none-any.whl (66 kB)\n",
            "\u001b[K     |████████████████████████████████| 66 kB 1.1 MB/s \n",
            "\u001b[?25hCollecting pycodestyle<2.10.0,>=2.9.0\n",
            "  Downloading pycodestyle-2.9.1-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[K     |████████████████████████████████| 41 kB 151 kB/s \n",
            "\u001b[?25hCollecting importlib-metadata<4.3,>=1.1.0\n",
            "  Downloading importlib_metadata-4.2.0-py3-none-any.whl (16 kB)\n",
            "Collecting mccabe<0.8.0,>=0.7.0\n",
            "  Downloading mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.3,>=1.1.0->flake8) (3.10.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.3,>=1.1.0->flake8) (4.1.1)\n",
            "Installing collected packages: pyflakes, pycodestyle, mccabe, importlib-metadata, flake8\n",
            "  Attempting uninstall: pycodestyle\n",
            "    Found existing installation: pycodestyle 2.10.0\n",
            "    Uninstalling pycodestyle-2.10.0:\n",
            "      Successfully uninstalled pycodestyle-2.10.0\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 4.13.0\n",
            "    Uninstalling importlib-metadata-4.13.0:\n",
            "      Successfully uninstalled importlib-metadata-4.13.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "markdown 3.4.1 requires importlib-metadata>=4.4; python_version < \"3.10\", but you have importlib-metadata 4.2.0 which is incompatible.\n",
            "gym 0.25.2 requires importlib-metadata>=4.8.0; python_version < \"3.10\", but you have importlib-metadata 4.2.0 which is incompatible.\u001b[0m\n",
            "Successfully installed flake8-5.0.4 importlib-metadata-4.2.0 mccabe-0.7.0 pycodestyle-2.9.1 pyflakes-2.5.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "importlib_metadata"
                ]
              }
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBg-8Bzm63d_"
      },
      "source": [
        "# Data\n",
        "\n",
        "Below I demonstrate a number of different approaches to getting data into our processing pipeline.  I mostly load data and common python modules from my GitHub account (https://github.com/todnewman).  I tend to store models and other artifacts of this workflow in my Google Drive.  When using this workflow, the developer will of course read from and save to their own Google Drive.  You'll need a Google account (which you probably need anyway to use this Notebook) and would have to authenticate.  That is all handled in number 2 below.\n",
        "\n",
        "Right now this workflow is automated and the products are very available.  This is very useful for using as training modules.\n",
        "\n",
        "##Methods of Data Insertion:\n",
        "1. Direct import from your PC local storage\n",
        "2. Load from Google Drive \n",
        "3. Load from GitHub\n",
        "4. Direct acquisition from repositories (like UC Irvine's Open Data Repository).  Use wget or import a URL directly in to pandas read_csv()."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fKRtNbcXhSzh"
      },
      "source": [
        "### Here's how you can load files to and from YOUR Google Drive.  \n",
        "\n",
        "It will give you a link to follow which will ask you to log in to your Google account.  Then your Google drive will show up as /content/drive/'My Drive'\n",
        "\n",
        "This is a good way to get larger datafiles into your workflows.  They'll take up space on your Google Drive, but you have 15GB for free.\n",
        "\n",
        "If you want to grab tod's lightning_sensor_data.csv, it is public and accessible at this link:  https://drive.google.com/file/d/12rb8lkz16iTWuvB2OVXwZ0SJvE8AYSps/view?usp=sharing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7joL7CqgJjQu",
        "outputId": "0a1e593e-e0ad-4834-d051-51a50aa0e21d"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd /content/drive/'My Drive'/models\n",
        "%ls\n",
        "%cp rain_model_mlp /content/rain_model_mlp"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n",
            "/content/drive/My Drive/models\n",
            " 1_id_confusion_matrix.png                 lunarlander_ac3.zip\n",
            " 1_rain_confusion_matrix.png               model_150_1000_30_36.hdf5\n",
            " 2_id_confusion_matrix.png                 model_150_1000_64_44.hdf5\n",
            " 2_rain_confusion_matrix.png               model_300_500_30_33\n",
            " annotations.tar.gz                        model_5_2.hdf5\n",
            " annotations.tar.gz.1                      model.hdf5\n",
            " basic_cnn_fmnist.hdf5                     model_lanl.hdf5\n",
            " checkpoint                                model_n17.hdf5\n",
            " cnn_fashion_mnist                         ngrok\n",
            " cnn_fashion_mnist_4                       ngrok-stable-linux-amd64.zip\n",
            " cnn_fashion_mnist_4.data-00000-of-00001   ngrok-stable-linux-amd64.zip.1\n",
            " cnn_fashion_mnist_4.index                 quake_model_mlp\n",
            " cnn_fashion_mnist_5.data-00000-of-00001   quake_model_mlp.41\n",
            " cnn_fashion_mnist_5.index                 quake_model_mlp.75\n",
            " cnn_fashion_mnist.data-00000-of-00001     quake_model_mlp.bak\n",
            " cnn_fashion_mnist.data-00000-of-00002     quake_model_mlp_done\n",
            " cnn_fashion_mnist.data-00001-of-00002     rain_model\n",
            " cnn_fashion_mnist.index                   rain_model_10k_dropout.2\n",
            "'Copy of lunarlander_ac3.zip'              rain_model_5k_dropout.3\n",
            " densenet_fmnist.hdf5                      rain_model_5k_dropout.4\n",
            " id_model                                  rain_model_mlp\n",
            "'id_model (1)'                             rain_model_mlp1\n",
            " images.tar.gz                             rain_model_mlp_1.hdf5\n",
            " images.tar.gz.1                           rain_model_mlp2\n",
            " lander_ac3_almost_trained2.gif            rain_model_mlp3\n",
            " lander_ac3_almost_trained.gif             rain_model_mlp4\n",
            " lander_ac3_partially_trained.gif          rain_model_mlp_bce\n",
            " lander_ac3_training.gif                   rain_model_mlp.hdf5\n",
            " lander.gif                                rain_seq_model\n",
            " lanl_lstm_model                           siamese_model_1.h5\n",
            " LANL_new_model                            siamese_model_1.hdf5\n",
            " LANL_new_model.05                         siamese_model.hdf5\n",
            " lanl_quake_lstm                           submission.csv\n",
            " lanl_quake_lstm.1                         tod_cnn_fmnist.hdf5\n",
            " lanl_quake_lstm150                        tod_unet.h5\n",
            " lanl_quake_lstm.2                         unet_demo_model.h5\n",
            " \u001b[0m\u001b[01;34mlog\u001b[0m/                                      \u001b[01;34mvideo\u001b[0m/\n",
            " lstm_model.h5                             working_model2.h5\n",
            " lstm_test.csv                             working_model3.h5\n",
            " lstm_train.csv                            working_model4.h5\n",
            " lunarlander                               working_model5.h5\n",
            " lunarlander_ac3_trained.pkl               working_model.h5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5LSY7qL2nhWs"
      },
      "source": [
        "### Loading data from GitHub\n",
        "\n",
        "Here's another really good way to get example data in.  I have a data project in GitHub that is associated with the coe_training project.  We can download the whole folder with the git clone command or just one file with the wget command."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NaNoF346iPSq",
        "outputId": "cb52aa35-ca50-44d5-9eed-b6ff986a61f4"
      },
      "source": [
        "%cd /content\n",
        "# Below is how you clone the whole data project from Tod's GitHub\n",
        "#!git clone https://github.com/todnewman/data\n",
        "# Below is how you can grab just the single .csv file for this project.  You can get this address by clicking on a file, then clicking 'RAW'\n",
        "#  in GitHub.  Then copy the link address from the browser.\n",
        "!wget -O /content/lightning_sensor_data.csv 'https://raw.githubusercontent.com/todnewman/data/master/lightning_sensor_data.csv'\n",
        "!wget -O /content/lightning_sensor_data_predict.csv 'https://raw.githubusercontent.com/todnewman/data/master/lightning_sensor_data_predict.csv'\n",
        "!wget -O /content/pytools.py 'https://raw.githubusercontent.com/todnewman/coe_training/master/pytools.py'\n",
        "import pytools\n",
        "tensorboard = False\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "--2022-11-29 04:48:43--  https://raw.githubusercontent.com/todnewman/data/master/lightning_sensor_data.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 670206 (654K) [text/plain]\n",
            "Saving to: ‘/content/lightning_sensor_data.csv’\n",
            "\n",
            "/content/lightning_ 100%[===================>] 654.50K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2022-11-29 04:48:43 (15.1 MB/s) - ‘/content/lightning_sensor_data.csv’ saved [670206/670206]\n",
            "\n",
            "--2022-11-29 04:48:43--  https://raw.githubusercontent.com/todnewman/data/master/lightning_sensor_data_predict.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 77587 (76K) [text/plain]\n",
            "Saving to: ‘/content/lightning_sensor_data_predict.csv’\n",
            "\n",
            "/content/lightning_ 100%[===================>]  75.77K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2022-11-29 04:48:43 (5.82 MB/s) - ‘/content/lightning_sensor_data_predict.csv’ saved [77587/77587]\n",
            "\n",
            "--2022-11-29 04:48:43--  https://raw.githubusercontent.com/todnewman/coe_training/master/pytools.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 8563 (8.4K) [text/plain]\n",
            "Saving to: ‘/content/pytools.py’\n",
            "\n",
            "/content/pytools.py 100%[===================>]   8.36K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-11-29 04:48:44 (66.0 MB/s) - ‘/content/pytools.py’ saved [8563/8563]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2V6oOFPGxJWI"
      },
      "source": [
        "## Connect Tensorboard\n",
        "\n",
        "Note that this gives a link to the Tensorboard URL at the end.  Train the model first and then click the  link below (will have ngrok.io at the end) to see results in Tensorboard.\n",
        "\n",
        "Here's alink with basic info on Tensorboard:  https://www.tensorflow.org/guide/summaries_and_tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TE_ENXwa22pK"
      },
      "source": [
        "if not tensorboard:\n",
        "    # Load the TensorBoard notebook extension\n",
        "    %load_ext tensorboard\n",
        "tensorboard = True"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r-PvkeFqs0Ex"
      },
      "source": [
        "## Split Data into Training and Test Sets\n",
        "This function takes a dataframe, the size of the test set in percentage of the whole, a randomizing seed, and the target column number as arguments.\n",
        "First we capture the target column to prevent it from being scaled by the scaler.\n",
        "Then, we convert the data (X) into numpy arrays and scale.  There are three scalers that are effective for this work.  MaxAbsScaler is better with sparse data.\n",
        "Finally, we use Python sklearn's train_test_split function to return X and y for training and X and y for test.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fmffmRHDsrVm"
      },
      "source": [
        "\n",
        "def split_data(df, size, predict_flag, target):\n",
        "    ''' Split the data into training and testing sets.  Column headers below are custom for the \"Douglas data set\".\n",
        "    NaN's are filled with zero to maintain consistent data shapes.\n",
        "    '''\n",
        "    from sklearn.model_selection import train_test_split\n",
        "    from sklearn import preprocessing\n",
        "    from imblearn.combine import SMOTEENN\n",
        "    from imblearn.over_sampling import ADASYN\n",
        "    \n",
        "    y = df.pop(target) # Remove the 'target' column that has the labels\n",
        "    y = np.array(y).astype(int) # in case the target is a float for some reason\n",
        "    X = df.fillna(0)    \n",
        "    \n",
        "    #  Scale the data.  This allows many of the algorithms to work more effectively\n",
        " \n",
        "    X = preprocessing.MinMaxScaler().fit_transform(X)\n",
        "    X = np.array(X)\n",
        "    \n",
        "    \n",
        "    \n",
        "    if predict_flag:\n",
        "        return X, y\n",
        "    else:\n",
        "        # Scale the data so it works better with the ML algorithms\n",
        "        X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=size) # Keeping the test set small to evaluate timing\n",
        "        #X_train = tf.convert_to_tensor(X_train)\n",
        "        print(\"X_train size: {}\".format(X_train.shape))\n",
        "        print(\"y_train size: {}\".format(y_train.shape))\n",
        "        print(\"X_test size: {}\".format(X_test.shape))\n",
        "        print(\"y_test size: {}\".format(y_test.shape))\n",
        "        \n",
        "        #sm = ADASYN(random_state=41)\n",
        "        sm = SMOTEENN(random_state=41)\n",
        "        X_train, y_train = sm.fit_resample(X_train, y_train) # This just oversamples the training set.    \n",
        "        return X_train, X_test, y_train, y_test"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cX_2N05eCWDA"
      },
      "source": [
        "## Create Multi-Layer Perceptron network for this problem\n",
        "\n",
        "We're designing this MLP network to work with our data shape.  Adding a Leaky RELU activation function and Dropout to control for overfitting.  Classification layer is a dense layer size one since we're only doing binary classification.\n",
        "\n",
        "* It may be interesting to play with the dropout parameters.  Sometimes I see better performance when dropout is higher at lower layer numbers and vice versa.\n",
        "* The Leaky Relu does really seem to be advantageous.  What is the optimal alpha parameter?  See here:  https://keras.io/layers/advanced-activations/\n",
        "* There are lots and lots of metrics to choose.  Maybe accuracy isn't always the best.  What other metrics (and loss functions) might be useful?  https://keras.io/metrics/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pJHGOJiIeFHE"
      },
      "source": [
        "def nn_mlp(input_shape, learning_rate):\n",
        "    model = Sequential()\n",
        "    model.add(Dense(64, input_dim=input_shape[1]))\n",
        "    model.add(LeakyReLU(alpha=0.05))\n",
        "    model.add(Dropout(0.6))\n",
        "    model.add(Dense(32))\n",
        "    model.add(LeakyReLU(alpha=0.05))\n",
        "    model.add(Dropout(0.6))\n",
        "    model.add(Dense(16))\n",
        "    model.add(LeakyReLU(alpha=0.05))\n",
        "    model.add(Dropout(0.6))\n",
        "    model.add(Dense(1, activation='sigmoid'))\n",
        "    optimizer = keras.optimizers.Adam(lr=learning_rate)\n",
        "    model.compile(loss='mse',\n",
        "              optimizer=optimizer,\n",
        "              metrics=['accuracy'])\n",
        "    return model"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLxcqNaRfzsv"
      },
      "source": [
        "## Process Data and execute shift_time function in order to create temporal supervised ML model.\n",
        "\n",
        "Shift_time essentially takes the 'shift_depth' number of previous sequences for each feature and makes new features out of them.  This flattens the record with new_features = orig_number_of_features * shift_depth\n",
        "\n",
        "Then shift_time takes the target vector and shifts the future target 'y_shift' sequences away into the record.  \n",
        "\n",
        "What this allows is one large record with the current samples of each feature, some number of previous samples for each feature (shift_depth number) and the target value from the future.\n",
        "\n",
        "In the case of this example, this allows us to use the shift_depth previous sequences of environmental data to predict whether it will be raining y_shift hours in the future."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 373
        },
        "id": "VFQlVueibb-z",
        "outputId": "2f79e0de-1f73-4aa7-8710-ea7d4163ad68"
      },
      "source": [
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from keras.layers import LSTM, Flatten, Dense, Activation, Dropout, MaxPooling1D\n",
        "from keras.models import Sequential\n",
        "from sklearn.metrics import cohen_kappa_score\n",
        "from keras.layers import LeakyReLU\n",
        "from numpy import concatenate\n",
        "from math import sqrt\n",
        "from keras.callbacks import TensorBoard\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "import pytools\n",
        "%load_ext pycodestyle_magic\n",
        "\n",
        "#\n",
        "#  Important parameters used to shift the data around\n",
        "#\n",
        "shift_depth = 8 # The number of previous observations to use as features\n",
        "y_shift = -12  # Shift the target so we're predicting events in the future.  In this case, 12 hours.\n",
        "\n",
        "\n",
        "target = 'Rain'\n",
        "num_epochs = 50\n",
        "batch_size = 64\n",
        "learning_rate = .001\n",
        "weights_file = 'rain_model_mlp'\n",
        "\n",
        "reuse_trained_model = True\n",
        "\n",
        "df = pd.read_csv('/content/lightning_sensor_data.csv', error_bad_lines=False)\n",
        "\n",
        "# convert series to supervised learning.  This is a sneaky trick that simplifies\n",
        "# data preparation by creating new features comprised of a sequence of features.\n",
        "# The number of features goes up when you do that by a factor of shift_depth.\n",
        "# We pull the label (df_y) out beforehand though, so past sequences of the label\n",
        "# don't get pulled in.\n",
        "\n",
        "def shift_time(data, df_y,  n_in=1,  y_shift=6, dropnan=True):\n",
        "    \"\"\"\n",
        "\t  Frame a time series as a supervised learning dataset. This relies heavily on the\n",
        "      Pandas .shift() method.\n",
        "\t  Arguments:\n",
        "\t\t  data: Sequence of observations as a list or NumPy array.\n",
        "\t\t  n_in: Number of lag observations as input (X).\n",
        "\t\t  dropnan: Boolean whether or not to drop rows with NaN values.\n",
        "\t  Returns:\n",
        "\t\t  Pandas DataFrame of series framed for supervised learning.\n",
        "\t  \"\"\"\n",
        "    n_vars = 1 if type(data) is list else data.shape[1]\n",
        "    df = pd.DataFrame(data)\n",
        "    cols, names = list(), list()\n",
        "    \n",
        "    # input sequence (t-n, ... t-1)\n",
        "    # This is where we select a sequence of data as features.\n",
        "    \n",
        "    for i in range(n_in, 0, -1):\n",
        "        cols.append(df.shift(i))\n",
        "        names += [('var%d(t-%d)' % (j+1, i)) for j in range(n_vars)]\n",
        "        \n",
        "    #  Here's where we shift our label.  My goal is to predict rain 12 hours in\n",
        "    #  the future using the last 8 values in the sequence for each of the X features.\n",
        "    #\n",
        "    y_len = df_y.shape[0]\n",
        "    df_y = df_y.shift(periods=y_shift).fillna(0)\n",
        "    df_y = df_y.drop(df_y.index[y_len-y_shift:y_len]) # drop rows that were shifted up\n",
        "    \n",
        "              \n",
        "    # put it all together using Pandas Concat\n",
        "    agg = pd.concat(cols, axis=1)\n",
        "    agg_len = agg.shape[0]\n",
        "    agg = agg.drop(agg.index[agg_len-y_shift:agg_len])\n",
        "    agg.columns = names\n",
        "    \n",
        "    # drop rows with NaN values\n",
        "    \n",
        "    if dropnan:\n",
        "        agg.dropna(inplace=True)\n",
        "        \n",
        "    agg = agg.reset_index(drop=True)\n",
        "    agg['Rain'] = df_y\n",
        "    return (agg)\n",
        "      \n",
        "#selecting only the column that we are going to use in the prediction process\n",
        "data_to_use=df[['Full Light Spectrum', 'Humidity (%)', 'IR Spectrum', 'Luminosity (lux)', 'Pressure (HP-900)', 'Temperature (deg F)', 'Rain']]\n",
        "\n",
        "# load dataset\n",
        "y = data_to_use.pop('Rain')\n",
        "dataset = data_to_use\n",
        "values = dataset.values\n",
        "\n",
        "# frame as supervised learning\n",
        "reframed = shift_time(values, y, shift_depth, y_shift)\n",
        "reframed['Rain'] = y\n",
        "\n",
        "reframed.to_csv(\"/content/reframed.csv\")\n",
        "\n",
        "# split into train and test sets\n",
        "\n",
        "train_X, test_X, train_y, test_y = split_data(reframed, 0.30, False, target) # Predict_flag = False\n",
        "\n",
        "# design network\n",
        "\n",
        "model = nn_mlp(train_X.shape, learning_rate)\n",
        "\n",
        "if reuse_trained_model:\n",
        "    model.load_weights(weights_file) \n",
        "    \n",
        "# fit network\n",
        "print()\n",
        "print(\"Fitting the Model.  See time below:\")\n",
        "\n",
        "model_name = 'rain_model_mlp.hdf5'\n",
        "\n",
        "cb = [ModelCheckpoint(model_name, monitor='val_accuracy', save_best_only=True, period=1)]\n",
        "\n",
        "# Call TensorBoard for visual metrics\n",
        "if tensorboard:\n",
        "    tbCallBack = keras.callbacks.TensorBoard(log_dir='/content/logs', \n",
        "                                             histogram_freq=1,\n",
        "                                             write_graph=True,\n",
        "                                             write_grads=True,\n",
        "                                             batch_size=batch_size,)\n",
        "\n",
        "# Fit the model based on everything we've defined to date.  Write metrics to \n",
        "# TensorBoard\n",
        "\n",
        "if tensorboard:\n",
        "    history = model.fit(train_X, train_y,            \n",
        "                              batch_size=batch_size,\n",
        "                              epochs=num_epochs,\n",
        "                              verbose=1,\n",
        "                              validation_data=(test_X, test_y), \n",
        "                              callbacks=[tbCallBack, cb],\n",
        "                              shuffle = False) \n",
        "else:\n",
        "    history = model.fit(train_X, train_y,            \n",
        "                              batch_size=batch_size,\n",
        "                              epochs=num_epochs,\n",
        "                              verbose=1,\n",
        "                              validation_data=(test_X, test_y),\n",
        "                              callbacks=cb,\n",
        "                              shuffle = False) \n",
        "\n",
        "\n",
        "score = model.evaluate(test_X, test_y, batch_size=batch_size)\n",
        "print(score)\n",
        "print(history)\n",
        "print(model.summary())\n",
        "\n",
        "model.save_weights(weights_file, overwrite=True)\n",
        "\n",
        "# plot history\n",
        "plt.plot(history.history['loss'], label='train')\n",
        "plt.plot(history.history['val_loss'], label='test')\n",
        "plt.legend()\n",
        "plt.show()\n",
        "\n",
        "\n",
        "df_out = pd.DataFrame()\n",
        "# make a prediction\n",
        "yhat = model.predict(test_X)\n",
        "df_out['yhat'] = pd.Series(yhat.flatten())\n",
        "df_out['y'] = test_y\n",
        "\n",
        "print(\"Report that scores the ability of the model to predict the target on the test data:\\n\")\n",
        "\n",
        "yhat_int = df_out['yhat'].astype(int)\n",
        "print(metrics.classification_report(test_y, yhat_int))\n",
        "\n",
        "# The Kappa statistic (or value) is a metric that compares an Observed Accuracy with an \n",
        "# Expected Accuracy (random chance).  The kappa statistic can range between -1 and 1. \n",
        "# The maximum value means complete agreement; zero or lower means chance agreement.\n",
        "\n",
        "ck_score = cohen_kappa_score(test_y, yhat_int)\n",
        "print(\"Cohen-Kappa Score: %7.2f\" % ck_score)\n",
        "\n",
        "# Confusion Matrix\n",
        "\n",
        "class_names = [ 'Not Rain', 'Rain']\n",
        "pytools.call_confusion_matrix(test_y,yhat_int, class_names, 'rain_confusion_matrix.png', 5)\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-7-79a3c3a9c304>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSequential\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmetrics\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcohen_kappa_score\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvanced_activations\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mLeakyReLU\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mconcatenate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mmath\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0msqrt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'keras.layers.advanced_activations'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LE9DILDd6rZY"
      },
      "source": [
        "## Save trained Model for future use\n",
        "\n",
        "Need to have the Google Drive loaded (see above)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEaoRldOJy7d"
      },
      "source": [
        "%cd /content\n",
        "%ls\n",
        "#%cp reframed.csv /content/drive/'My Drive'/reframed.csv\n",
        "%cp rain_model_mlp.hdf5 /content/drive/'My Drive'/models/rain_model_mlp_1.hdf5"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3Uf96A5HHLm"
      },
      "source": [
        "# Predict New Data\n",
        "\n",
        "We're going to load our saved model first."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ne26SZIZi0TJ"
      },
      "source": [
        "from keras.models import load_model\n",
        "\n",
        "#weights_file = 'rain_model_mlp'\n",
        "#model.load_weights(weights_file)\n",
        "model = load_model(model_name)\n",
        "df1 = pd.read_csv('/content/lightning_sensor_data_predict.csv', error_bad_lines=False)\n",
        "print(df1.shape)\n",
        "\n",
        "df1 = df1[0:60]\n",
        "\n",
        "df_time = df1['Time']\n",
        "\n",
        "data_to_use1=df1[['Full Light Spectrum', 'Humidity (%)', 'IR Spectrum', 'Luminosity (lux)', 'Pressure (HP-900)', 'Temperature (deg F)', 'Rain']]\n",
        "\n",
        "dataset = data_to_use1\n",
        "\n",
        "# Keep this label but don't predict using it.  \n",
        "# We'll use it later to demonstrate the validity of our predictions.\n",
        "\n",
        "y = dataset.pop('Rain')\n",
        "\n",
        "values = dataset.values\n",
        "\n",
        "# frame as supervised learning\n",
        "\n",
        "reframed = shift_time(values, y, shift_depth, y_shift)\n",
        "\n",
        "#print(reframed)\n",
        "\n",
        "predict_X, y_bogus = split_data(reframed, 0.30, True, target) # Predict_flag = False\n",
        "\n",
        "# reshape input to be 3D [samples, timesteps, features]\n",
        "\n",
        "print(predict_X.shape)\n",
        "\n",
        "probs = model.predict(predict_X)\n",
        "print (type(probs))\n",
        "df_t = pd.Series(df_time[0:predict_X.shape[0]])\n",
        "\n",
        "#df_pred = reframed.copy()\n",
        "df_pred = pd.DataFrame()\n",
        "df_pred['Time'] = df_t\n",
        "\n",
        "#df_pred['Predicted Value'] = (predict_val[0:])\n",
        "df_pred['Predicted Probability of Rain'] = probs\n",
        "df_pred['Actual Rain Event'] = y\n",
        "\n",
        "#df_pred.to_csv(out_filename)\n",
        "k = df_pred.keys()\n",
        "\n",
        "#df_pred.sort_values(['Predicted Value'], ascending=[False])\n",
        "\n",
        "plt.figure( figsize=(20,10))\n",
        "plt.title(\"Probability of rain in 12 hours\")\n",
        "plt.ylabel(\"Probability of Rain\")\n",
        "\n",
        "plt.fill_between(df_pred[k[0]], df_pred[k[1]],  alpha=0.4)\n",
        "plt.fill_between(df_pred[k[0]], df_pred[k[2]],  alpha=0.4)\n",
        "\n",
        "plt.plot(df_pred[k[0]], df_pred[k[1]], label=\"Probability of Rain in 12 hours\")\n",
        "plt.plot(df_pred[k[0]], df_pred[k[2]], label=\"Actual Rain Event\")\n",
        "plt.legend(loc='best')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6SHib6xxfR6T"
      },
      "source": [
        "%cd /content/logs/train\n",
        "%tensorboard --logdir ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJejdhw5s7Zl"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}