Basic_Data_Engineering.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Basic Data Engineering.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/todnewman/coe_training/blob/master/Basic_Data_Engineering.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cHLkhLc7mlr7"
      },
      "source": [
        "# Basic Pandas Data Engineering\n",
        "Author: W. Tod Newman\n",
        "\n",
        "## Learning Objectives\n",
        "\n",
        "\n",
        "*   Learn how to import files into Pandas Dataframes and how to develop understanding of what's in the data\n",
        "*   Learn how to access and manipulate the data in the DataFrame\n",
        "*   Learn how to do vectorized math and Boolean Filtering on the Dataframe.\n",
        "*   Understand the best ways to do looping in Python\n",
        "*   Understand some simple approaches to data ingest and how they compare.\n",
        "\n",
        "In this exercise we're using data from the UC Irvine Open Data archive.  This data is in .CSV format and is easily pulled into Pandas.  See the Handling Complex Data files notebook for details on how to manipulate larger, zipped files from UCI.\n",
        "\n",
        "## Contents\n",
        "* [Basic Pandas Data Operations](#scrollTo=gk7FpEzVbu7X)\n",
        "* [Python Loop Optimization Demonstration](#scrollTo=keF9jUSWOVVf)\n",
        "* [Python Data Ingest Optimization for Speed and Memory Footprint](#scrollTo=zRYJZ1EkI4Nr&line=16&uniqifier=1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H7o8tWaWmmTu"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import time\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "pd.set_option('display.max_rows', 500)\n",
        "pd.set_option('display.max_columns', 500)\n",
        "pd.set_option('display.width', 1000)\n",
        "!pip install line_profiler"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gk7FpEzVbu7X"
      },
      "source": [
        "# Basic Pandas Operations\n",
        "## Import a Data File and Read it Into a Pandas DataFrame\n",
        "\n",
        "Here  we pull data from the UC Irvine open data repository.  This data is useful because often it can be used to test algorithms and ML workflows.  Plus, it works well in CoLaboratory because when we load it into the Google CoLaboratory system it doesn't need to pass through the firewall.\n",
        "\n",
        "First we read the file in as a .CSV and then we print the dataframe.  You can print the dataframe by merely typing \"df\" only if you do it on the last line.  Otherwise, you need to print(df).  The formatting of the latter will be less pleasant than the former, BTW.\n",
        "\n",
        "You can comment out any of the last 4 lines to see what those lines do.  The last one uncommented will be printed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jH7cg8uw0N0C"
      },
      "source": [
        "# Get COVID-19 Files from JHU GitHub\n",
        "!wget -O time_series_19-covid-Confirmed.csv https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv\n",
        "!wget -O time_series_19-covid-Deaths.csv https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv\n",
        "!wget -O time_series_19-covid-Recovered.csv https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv\n",
        "\n",
        "# Grab Data from Tod's GitHub\n",
        "!wget -O world_pop_data.csv https://raw.githubusercontent.com/todnewman/data/master/covid19/world_pop_data.csv\n",
        "!wget -O china_pop_data.csv https://raw.githubusercontent.com/todnewman/data/master/covid19/china_pop_data.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I3_aCJFzfeAq"
      },
      "source": [
        "## Read Parkinson's Data into a Pandas Dataframe\n",
        "\n",
        "We'll use this data to demonstrate some basic capabilities of Pandas.  Then we'll show how it all comes together using the COVID-19 Data we just downloaded in the previous step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hF1VAF05FZUU"
      },
      "source": [
        "# Grab Data from UC Irvine archives.  Do some basic Pandas actions.\n",
        "width = 20\n",
        "height = 10\n",
        "df = pd.read_csv(r'https://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/parkinsons.data')\n",
        "\n",
        "# Misc things you can do with a DataFrame\n",
        "\n",
        "#df.tail(2) # Last 2 records\n",
        "#df.keys() # List of all the Columns\n",
        "#print(df['name']) # The column with the header 'name'\n",
        "\n",
        "# Grab a specific Value from the DataFrame\n",
        "\n",
        "row_indexer = 19\n",
        "column_indexer = 12\n",
        "print(df.iloc[row_indexer,column_indexer])\n",
        "\n",
        "# All the values in column 12\n",
        "\n",
        "#print(df.iloc[:,12])  \n",
        "\n",
        "# Records/Rows 0 through 10\n",
        "\n",
        "#print(df[10:40] )\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# Plot all the parameters\n",
        "\n",
        "'''df.plot(figsize=(width,height))\n",
        "plt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))\n",
        "plt.tight_layout()\n",
        "plt.show()'''\n",
        "\n",
        "df.head() # First 5 records"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ElyJB1N4cxly"
      },
      "source": [
        "## Print Column Headers (keys) and Use for Quick Data Exploration\n",
        "\n",
        "This is a useful thing to do.  If you assign df.keys() to a variable, it becomes a Python list.  You can iterate through the list to access the individual keys one by one or you can access each parameter in the list with it's index (i.e., keys[3]).  This is a useful way to select features without having to know the name of the header."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_hEsGg_yHql8"
      },
      "source": [
        "keys = df.keys()\n",
        "print (keys)\n",
        "print()\n",
        "print(keys[3]) # Handy in case you don't want to type the name\n",
        "\n",
        "# iterate through the keys.  Here's a useful way to do data exploration.\n",
        "for i,k in enumerate(keys):\n",
        "    if i>0:\n",
        "        df.plot( y=keys[i])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sqTCb6E-pb90"
      },
      "source": [
        "## Information on the DataFrame\n",
        "\n",
        "Pandas gives us a couple of useful calls to get info on the different features in the DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3DnH1Y2YLCs"
      },
      "source": [
        "print (df.info())\n",
        "print (df.describe())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mUJ2IEncpvut"
      },
      "source": [
        "## Accessing parameters in the DataFrame\n",
        "\n",
        "You can see from the below how Pandas gives access to the index, columns, and values of the DataFrame.  You can use .columns to change the header names if that's desirable.  Understanding the shape of the dataframe (in this case, it is 195 rows by 24 columns) will help you understand if you get shape mismatch errors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bqjYzEOXYbOC"
      },
      "source": [
        "print (df.index)\n",
        "print (\"Shape of the Dataframe is: \", df.shape)\n",
        "\n",
        "print (df.columns)  # Same thing as df.keys()\n",
        "\n",
        "print (df.values)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_rz44adGquCE"
      },
      "source": [
        "## Sorting\n",
        "\n",
        "sort_values is useful.  You can either sort by something like I did here (keys[3]) or you can sort by the header name ('MDVP:Flo(Hz)').  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XdI4ci5PYoAO"
      },
      "source": [
        "keys = df.keys()\n",
        "\n",
        "print (keys[1:6])\n",
        "\n",
        "df.sort_values(keys[6], ascending=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zMoEuG5rOIJ"
      },
      "source": [
        "## Print out selected Columns in the DataFrame\n",
        "\n",
        "This allows you to pick which columns you want to see.  The first line selects just one column.  The second shows how you can select a slice (multiple contiguous columnts). The third shows how you can just select discrete sets of columns.  The fourth line shows us passing an array of headers to the DataFrame.  You will probably notice that this isn't much different than the approach for the 3rd line."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yy0FYthaaD0q"
      },
      "source": [
        "#df[keys[1]]\n",
        "#df[keys[1:6]]\n",
        "#df[['MDVP:Fo(Hz)', 'MDVP:Jitter(%)']]\n",
        "my_header_list = ['MDVP:Fo(Hz)', 'MDVP:Jitter(%)', 'MDVP:PPQ']\n",
        "df2 = df[my_header_list]\n",
        "df2.to_csv('three_cols.csv')\n",
        "df2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Pf2zauV0qqD"
      },
      "source": [
        "!ls -l /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V1TC8jtEaR1e"
      },
      "source": [
        "#df[2:8] # Filter out 3 rows in the DataFrame\n",
        "df.loc[:,[keys[1], keys[7]]] # Filter headers 1 and 7 for all records\n",
        "#df.loc[1:4,[keys[1], keys[7]]] # Filter headers 1 and y for just three records"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xh-RDgais6q9"
      },
      "source": [
        "## Filtering\n",
        "\n",
        "Often its useful to be able to select all records that meet a criteria.  For example, we might want all records where MDVP:Fo(Hz) > 120.  Or as another example, we might all records where YEAR == 2018."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i_1MDiPlafIs"
      },
      "source": [
        "%%time\n",
        "mdvp_hi_fo_freq = df[keys[1]] < 120  # Set up the boolean.  The vaue of mdvp_hi_fo_freq will be True if it is > 120 and False if it is < 120\n",
        "mdvp_hi_fo_freq2 = df[keys[1]] >= 117\n",
        "df_117_120 = (df[mdvp_hi_fo_freq&mdvp_hi_fo_freq2])\n",
        "print(df_117_120)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QmM4fJQIwrfl"
      },
      "source": [
        "## Pandas Math"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QO5hi_pfwmCf"
      },
      "source": [
        "\n",
        "print(keys)\n",
        "print(\"Mean Value for Column 1 is %s\" % df[keys[1]].mean())  # Calculate the mean value for the whole column\n",
        "print(\"Median Value for Column 1 is %s\" % df[keys[1]].median())\n",
        "#print(\"Cumulative Sum for column 1 is %s\" % df[keys[1]].apply(np.cumsum))\n",
        "df['Product1'] = df[keys[1]] * df[keys[2]]  #Multiply two DataFrame columns into a product.\n",
        "print(df.keys())\n",
        "df['Product1'][0:5]\n",
        "\n",
        "#df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRCoBhu60wnh"
      },
      "source": [
        "## Putting it all together using COVID-19 Data!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uW5EDSga0viR"
      },
      "source": [
        "#\n",
        "#  First import the csv files into Pandas DataFrames\n",
        "#\n",
        "\n",
        "df_confirmed_w = pd.read_csv('time_series_19-covid-Confirmed.csv')\n",
        "df_deaths_w = pd.read_csv('time_series_19-covid-Deaths.csv')\n",
        "df_recovered_w = pd.read_csv('time_series_19-covid-Recovered.csv')\n",
        "\n",
        "pop_df = pd.read_csv('world_pop_data.csv')\n",
        "china_pop_df = pd.read_csv('china_pop_data.csv')\n",
        "\n",
        "# Fix dates to remove slashes and replace with underscores.  Makes things\n",
        "# simpler.\n",
        "\n",
        "df_confirmed_w.columns = df_confirmed_w.columns.str.replace(\"/\", \"_\")\n",
        "df_deaths_w.columns = df_deaths_w.columns.str.replace(\"/\", \"_\")\n",
        "df_recovered_w.columns = df_recovered_w.columns.str.replace(\"/\", \"_\")\n",
        "\n",
        "df_keys = df_confirmed_w.keys()\n",
        "\n",
        "# Simple way to grab today's date, because it will be the header for the last\n",
        "# column.\n",
        "\n",
        "col_today = df_keys[len(df_keys) - 1]\n",
        "col_yesterday = df_keys[len(df_keys) - 2]\n",
        "col_two_days = df_keys[len(df_keys) -3]\n",
        "print(f\"Today's date: {col_today}\")\n",
        "\n",
        "#df_confirmed_w.sort_values(by=col_today, ascending=False)[0:20]\n",
        "df_confirmed_w['Country-Prov'] = df_confirmed_w[['Country_Region', 'Province_State']].fillna('').sum(axis=1)\n",
        "df_recovered_w['Country-Prov'] = df_recovered_w[['Country_Region', 'Province_State']].fillna('').sum(axis=1)\n",
        "df_deaths_w['Country-Prov'] = df_deaths_w[['Country_Region', 'Province_State']].fillna('').sum(axis=1)\n",
        "\n",
        "#\n",
        "# Blend Population for Chinese Provinces into the dataset\n",
        "#\n",
        "df_confirmed_w = df_confirmed_w.merge(china_pop_df, how='left', left_on=['Country_Region', 'Province_State'], \n",
        "                  right_on = ['Country', 'Province'])\n",
        "df_deaths_w = df_deaths_w.merge(china_pop_df, how='left', left_on=['Country_Region', 'Province_State'], \n",
        "                  right_on = ['Country', 'Province'])\n",
        "#\n",
        "# Blend Population for World Countries into the dataset\n",
        "#\n",
        "df_confirmed_w = df_confirmed_w.merge(pop_df, how='left', left_on=['Country_Region'], \n",
        "                  right_on = ['name'])\n",
        "df_deaths_w = df_deaths_w.merge(pop_df, how='left', left_on=['Country_Region'], \n",
        "                  right_on = ['name'])\n",
        "\n",
        "#\n",
        "# Do some math to make sure populations are aligned across countries and provinces\n",
        "#\n",
        "df_deaths_w['pop'] = df_deaths_w['pop2019'].fillna(df_deaths_w['chipop'])\n",
        "df_deaths_w.drop(['pop2019','chipop'], axis=1, inplace=True)\n",
        "df_confirmed_w['pop'] = df_confirmed_w['pop2019'].fillna(df_confirmed_w['chipop'])\n",
        "df_confirmed_w.drop(['pop2019','chipop'], axis=1, inplace=True)\n",
        "\n",
        "#\n",
        "# ********** SHIFT TO BUILDING TODAY'S DATAFRAME ********************\n",
        "#\n",
        "\n",
        "\"\"\"\n",
        "Calculate the instantaneous rate of change.  This is the tangent to the curve\n",
        "and provides us today's slope of a non-linear function (Case Rates, Death Rates, etc.).\n",
        "To do this we're going to fit a 3rd order polynomial to the last 60 data points\n",
        "and then use that to calculate the IROC.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "def g(x, z, h = 1e-3): # This does the calculus\n",
        "    def function(x):\n",
        "        y = np.polyval(z,x)\n",
        "        return y\n",
        "    deriv = (function(x + h) - function(x)) / h\n",
        "    return round(deriv, 4)\n",
        "    \n",
        "def calc_fit(row): # Calculate the first derivative (IROC)\n",
        "    # We're going to look at the latest 60 results for fitting our curve.  This\n",
        "    # will help us avoid weirdness like negative rates.\n",
        "    \n",
        "    row = row.sort_values(ascending=True)\n",
        "    lenrow = len(row)\n",
        "    row = row[lenrow-60: lenrow]\n",
        "\n",
        "    # New length of the series of values for our curve fit\n",
        "    axisvalues=list(range(1,len(row)+1))\n",
        "    len_d = len(axisvalues)\n",
        "    \n",
        "    # Want at least 3 points to fit a curve\n",
        "    if len_d > 3: # Enough points to do a reasonable LR fit\n",
        "        z = np.polyfit(axisvalues, row, 3)\n",
        "    else:\n",
        "        z = (0,0,0)\n",
        "    \n",
        "    # Now that we have fit a curve, take the derivatives\n",
        "    iroc = g(len_d, z)\n",
        "    return iroc\n",
        "\n",
        "def calc_fit_delta(row): # Calculate a simple 2nd derivative (dIROC)\n",
        "    # We're going to look at the latest 60 results for fitting our curve.  This\n",
        "    # will help us avoid weirdness like negative rates.\n",
        "\n",
        "    row = row.sort_values(ascending=True)\n",
        "    lenrow = len(row)\n",
        "    row = row[lenrow-60: lenrow]\n",
        "\n",
        "    # New length of the series of values for our curve fit\n",
        "    axisvalues=list(range(1,len(row)+1))\n",
        "    len_d = len(axisvalues)\n",
        "    \n",
        "    if len_d > 3: # Enough points to do a reasonable LR fit\n",
        "        z = np.polyfit(axisvalues, row, 2)\n",
        "    else:\n",
        "        z = (0,0,0)\n",
        "        \n",
        "    #Take the derivative of today's value minus that of two days ago\n",
        "    iroc_d = g(len_d, z) - g(len_d-2, z)\n",
        "    return iroc_d\n",
        "#\n",
        "#  Calculate the curve slopes for death and confirmed case curves.\n",
        "#\n",
        "\n",
        "for i,nm in enumerate(['deaths', 'confirmed']):\n",
        "    if i==0:\n",
        "        df = df_deaths_w\n",
        "    else:\n",
        "        df = df_confirmed_w\n",
        "    \n",
        "    dfm_keys = df.keys()\n",
        "    today_idx = df.columns.get_loc(col_today)\n",
        "\n",
        "    # These are all the daily number features\n",
        "    day_features = dfm_keys[4:today_idx+1]\n",
        "\n",
        "    #df_t = df_merge_c[day_features]\n",
        "    # Normalize cases by 1000 population for slope\n",
        "    df_t_norm = df[day_features].div(df['pop'], axis=0)\n",
        "    df_t = df[day_features]\n",
        "    \n",
        "    axisvalues=list(range(1,len(df_t.columns)+1))\n",
        "\n",
        "    iroc = df_t.fillna(0).apply(calc_fit, axis=1)\n",
        "    iroc_d = df_t.fillna(0).apply(calc_fit_delta, axis=1)\n",
        "    iroc_n = df_t_norm.fillna(0).apply(calc_fit, axis=1)\n",
        "    iroc_d_n = df_t_norm.fillna(0).apply(calc_fit_delta, axis=1)\n",
        "    df['inst_rate_of_change'] = iroc\n",
        "    df['inst_roc_delta'] = iroc_d\n",
        "    df['inst_rate_of_change_norm'] = iroc_n\n",
        "    df['inst_roc_delta_norm'] = iroc_d_n\n",
        "\n",
        "#\n",
        "# Now we have enough to build Summary DataFrame for Current Date\n",
        "#\n",
        "df_summ_w = pd.DataFrame()\n",
        "\n",
        "df_summ_w['Country_Region'] = df_confirmed_w['Country_Region']\n",
        "df_summ_w['Province_State'] = df_confirmed_w['Province_State']\n",
        "df_summ_w['pop'] = df_confirmed_w['pop']\n",
        "df_summ_w['state'] = df_confirmed_w['state']\n",
        "df_summ_w['Country-Prov'] = df_confirmed_w['Country-Prov']\n",
        "df_summ_w['Lat'] = df_confirmed_w['Lat']\n",
        "df_summ_w['Long'] = df_confirmed_w['Long']\n",
        "df_summ_w['Confirmed'] = df_confirmed_w[col_today]\n",
        "df_summ_w['Deaths'] = df_deaths_w[col_today]\n",
        "df_summ_w['Recovered'] = df_recovered_w[col_today]\n",
        "df_summ_w['New_Cases'] = df_confirmed_w[col_today] - df_confirmed_w[col_yesterday]\n",
        "df_summ_w['Delta_Active'] = df_summ_w['New_Cases'] - (df_confirmed_w[col_yesterday] - df_confirmed_w[col_two_days])\n",
        "df_summ_w['Delta_Deaths'] = df_summ_w['Deaths'] - (df_deaths_w[col_yesterday])\n",
        "df_summ_w['IROC_c'] = df_confirmed_w['inst_rate_of_change']\n",
        "df_summ_w['dIROC_c'] = df_confirmed_w['inst_roc_delta']\n",
        "df_summ_w['IROC_d'] = df_deaths_w['inst_rate_of_change']\n",
        "df_summ_w['dIROC_d'] = df_deaths_w['inst_roc_delta']\n",
        "df_summ_w['IROC_c_n'] = df_confirmed_w['inst_rate_of_change_norm']\n",
        "df_summ_w['dIROC_c_n'] = df_confirmed_w['inst_roc_delta_norm']\n",
        "df_summ_w['IROC_d_n'] = df_deaths_w['inst_rate_of_change_norm']\n",
        "df_summ_w['dIROC_d_n'] = df_deaths_w['inst_roc_delta_norm']\n",
        "df_summ_w['cases_per_1000'] = (df_summ_w['Confirmed']/(df_summ_w['pop'])).fillna(0)\n",
        "df_summ_w['Deaths_per_1000'] = (df_summ_w['Deaths']/(df_summ_w['pop'])).fillna(0)\n",
        "df_summ_w['%Deaths_to_cases'] = (df_summ_w['Deaths']/(df_summ_w['Confirmed']+.001)*100).fillna(0)\n",
        "\n",
        "# We're only going to print the below columns to save space and make it readable.\n",
        "print_cols = ['Country-Prov','Confirmed', 'Deaths', 'Recovered', 'New_Cases',\n",
        "              'Delta_Deaths', 'IROC_c_n', 'dIROC_c_n', 'IROC_d_n', 'dIROC_d_n',\n",
        "              'cases_per_1000', 'Deaths_per_1000']\n",
        "caseno_fil = df_summ_w['New_Cases'] > 0\n",
        "df_summ_w[caseno_fil][print_cols].sort_values(by='IROC_c_n', ascending=False)[0:20]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CS6KmvPOSAdE"
      },
      "source": [
        "# How to select A Particular Country for Analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FegSg4qrIvr9"
      },
      "source": [
        "country = 'China'\n",
        "aus_fil = df_summ_w['Country_Region'] == country\n",
        "df_summ_w[aus_fil][print_cols].sort_values(by='IROC_c_n', ascending=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KnziAK-Z6yfl"
      },
      "source": [
        "df_summ_w[print_cols].sort_values(by='Deaths_per_1000', ascending=False)[0:25]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "keF9jUSWOVVf"
      },
      "source": [
        "# Efficient DataFrame Operations\n",
        "\n",
        "### Looping\n",
        "\n",
        "We'll show a few of approaches to looping and time them.  This example is derived from work at https://github.com/s-heisler/pycon2017-optimizing-pandas\n",
        "\n",
        "This function essentially calculates the Great Circle distance between two (Lat,Long) pairs.  We're going to get the distances between Tod's house (32.266, 110.985) and the location of every hotel in NYC.  \n",
        "\n",
        "### Haversine formula\n",
        "\n",
        "${\\displaystyle {\\begin{aligned}d&=2r\\arcsin \\left({\\sqrt {\\operatorname {hav} (\\varphi _{2}-\\varphi _{1})+\\cos(\\varphi _{1})\\cos(\\varphi _{2})\\operatorname {hav} (\\lambda _{2}-\\lambda _{1})}}\\right)\\\\&=2r\\arcsin \\left({\\sqrt {\\sin ^{2}\\left({\\frac {\\varphi _{2}-\\varphi _{1}}{2}}\\right)+\\cos(\\varphi _{1})\\cos(\\varphi _{2})\\sin ^{2}\\left({\\frac {\\lambda _{2}-\\lambda _{1}}{2}}\\right)}}\\right)\\end{aligned}}}$\n",
        "\n",
        "where\n",
        "\n",
        "φ1, φ2 are the latitude of point 1 and latitude of point 2 (in radians),\n",
        "λ1, λ2 are the longitude of point 1 and longitude of point 2 (in radians).\n",
        "\n",
        "### Basic setup block below"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-IxubbxfOc48"
      },
      "source": [
        "from math import radians, cos, sin, asin, sqrt\n",
        "\n",
        "!wget -O /content/NYC_hotels.csv 'https://raw.githubusercontent.com/todnewman/data/master/NYC_hotels.csv.txt'\n",
        "\n",
        "df = pd.read_csv('NYC_hotels.csv', encoding='cp1252')\n",
        "\n",
        "\n",
        "def haversine(lat1, lon1, lat2, lon2):\n",
        "    \"\"\"\n",
        "    Calculate the great circle distance between two points \n",
        "    on the earth (specified in decimal degrees)\n",
        "    \"\"\"\n",
        "    radius = 3958.8 # Radius of earth in miles\n",
        "    \n",
        "    # convert decimal degrees to radians.  Map applies the deg2rad\n",
        "    # function to the list of latitudes and longitudes.\n",
        "    lat1, lon1, lat2, lon2 = map(np.deg2rad, [lat1, lon1, lat2, lon2])\n",
        "    \n",
        "    # haversine formula \n",
        "    dlat = lat2 - lat1 \n",
        "    dlon = lon2 - lon1 \n",
        "\n",
        "    # Below we calculate the haversine distance using the formula above\n",
        "    a = np.sin(dlat/2)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2)**2\n",
        "    d = 2 * radius * np.arcsin(np.sqrt(a)) \n",
        "    \n",
        "    return d\n",
        "\n",
        "# Define a function to manually loop over all rows and return a series of distances\n",
        "# This is the Dumb Function.\n",
        "#\n",
        "def haversine_looping(df):\n",
        "    distance_list = []\n",
        "    for i in range(0, len(df)): # Go through every row in the dataset\n",
        "        # Find distance from Tod's house in Tucson to every Hotel in NYC\n",
        "        # we do this by walking down every row of hotels and grabbing lat and long.\n",
        "        d = haversine(32.266, 110.985, df.iloc[i]['latitude'], df.iloc[i]['longitude'])\n",
        "        distance_list.append(d)\n",
        "    return distance_list\n",
        "\n",
        "print (df.keys())\n",
        "print (df.info())\n",
        "\n",
        "res = []\n",
        "name = []\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kmSuk4PdSF64"
      },
      "source": [
        "### The WORST way to do it!\n",
        "\n",
        "Here we're looping through the dataframe one loop at a time.  This seems natural and is easy to follow, but it's also the slowest possible way to solve this problem because it does not take advantage of any Python optimizations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nwv_rdcwRYgx"
      },
      "source": [
        "res = []\n",
        "num = []\n",
        "mean_res = []\n",
        "std_res = []\n",
        "\n",
        "loops = 5\n",
        "\n",
        "for l in range(0,loops):\n",
        "    start = time.time()\n",
        "    #\n",
        "    # Run the haversine looping function by passing the entire dataframe to the \n",
        "    # looping function above.  \"For i in range(0,len(df)):\"\n",
        "    #\n",
        "    df['distance'] = haversine_looping(df)\n",
        "\n",
        "    # Measure the timing for this approach\n",
        "    end = time.time()\n",
        "    delta = (end - start)*1000\n",
        "    msg = (\"%7.3F ms\" % delta) \n",
        "    res.append(msg)\n",
        "    num.append(delta)\n",
        "name.append(\"Basic Looping\")\n",
        "print(f\"RESULTS of {loops} Loops - Basic Looping\")\n",
        "print(res)\n",
        "print(f\"Mean: {np.mean(num, axis=0)}, StdDev: {np.std(num, axis=0)}\")\n",
        "mean_res.append(np.mean(num, axis=0))\n",
        "std_res.append(np.std(num, axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEYw7HbbTuGO"
      },
      "source": [
        "### A Better Way...\n",
        "\n",
        "Here we're going to use .itterrows(), which is an optimized looping function.  This is a data generator which returns a row index and value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g3x7Btc3RZA3"
      },
      "source": [
        "res = []\n",
        "num = []\n",
        "\n",
        "# Haversine applied on rows via iteration\n",
        "for l in range(0,loops):\n",
        "    start = time.time()\n",
        "    haversine_series = []\n",
        "    for index, row in df.iterrows():\n",
        "        haversine_series.append(haversine(32.266, 110.985, row['latitude'], row['longitude']))\n",
        "    df['distance'] = haversine_series\n",
        "    end = time.time()\n",
        "    delta = (end - start)*1000\n",
        "    msg = (\"%7.3F ms\" % delta)  \n",
        "    res.append(msg)\n",
        "    num.append(delta)\n",
        "name.append('Iterrow Looping')\n",
        "print(f\"RESULTS of {loops} Loops - Optimized Looping\")\n",
        "print(res)\n",
        "print(f\"Mean: {np.mean(num, axis=0)}, StdDev: {np.std(num, axis=0)}\")\n",
        "mean_res.append(np.mean(num, axis=0))\n",
        "std_res.append(np.std(num, axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhPJBfV9UFut"
      },
      "source": [
        "### Better yet...\n",
        "\n",
        "Using the .apply() function is even better than .itterows()\n",
        "\n",
        ".apply() is more optimized than iterrows()\n",
        "\n",
        "We use an anonymous lambda function to apply our Haversine function on each row, which allows us to point to specific cells within each row as inputs to the function. The lambda function includes the axis parameter at the end, in order to specify whether Pandas should apply the function to rows (axis = 1) or columns (axis = 0)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QWgcGZxARaTl"
      },
      "source": [
        "res = []\n",
        "num = []\n",
        "\n",
        "# Haversine applied on rows via iteration\n",
        "for l in range(0,loops):\n",
        "    start = time.time()\n",
        "    # Timing apply on the Haversine function\n",
        "    df['distance'] = df.apply(lambda row: haversine(32.266, 110.985, row['latitude'], row['longitude']), axis=1) \n",
        "    end = time.time()\n",
        "    delta = (end - start)*1000\n",
        "    msg = (\"%7.3F ms\" % delta)  \n",
        "    res.append(msg)\n",
        "    num.append(delta)\n",
        "name.append('Looping with Apply')\n",
        "print(f\"RESULTS of {loops} Loops - Looping with Apply Function\")\n",
        "print(res)\n",
        "print(f\"Mean: {np.mean(num, axis=0)}, StdDev: {np.std(num, axis=0)}\")\n",
        "mean_res.append(np.mean(num, axis=0))\n",
        "std_res.append(np.std(num, axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "guQivdX3U0bG"
      },
      "source": [
        "# Lets use the line profiler to figure out what the lambda is doing and where we can optimize next!\n",
        "\n",
        "%load_ext line_profiler\n",
        "\n",
        "%lprun -f haversine df.apply(lambda row: haversine(32.266, 110.985, row['latitude'], row['longitude']), axis=1)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U7PBuQA3kv95"
      },
      "source": [
        "### Vectorization!!\n",
        "\n",
        "Vectorization is the process of executing operations on entire arrays at once.  This unlocks the power of the modern processing architectures.\n",
        "\n",
        "*  Pandas includes all sorts of vectorized functions for everything from mathematical operations to aggregations and string functions (for an extensive list of available functions, check out the Pandas docs). \n",
        "*  The built-in functions are optimized to operate specifically on Pandas series and DataFrames. \n",
        "\n",
        "Vectorized Pandas functions are going  almost always be preferable to looping techniques like we show above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ar4QU8A6RvqR"
      },
      "source": [
        "res = []\n",
        "num = []\n",
        "\n",
        "# Haversine applied on rows via iteration\n",
        "for l in range(0,loops):\n",
        "    start = time.time()\n",
        "    # Vectorized implementation of Haversine applied on Pandas series\n",
        "    df['distance'] = haversine(32.266, 110.985, df['latitude'].astype(float), df['longitude'].astype(float))\n",
        "    #results['Pandas Vectorization'] = _.best\n",
        "    end = time.time()\n",
        "    delta = (end - start)*1000\n",
        "    msg = (\"%7.3F ms\" % delta)  \n",
        "    res.append(msg)\n",
        "    num.append(delta)\n",
        "name.append('Pandas Vectorization')\n",
        "print(f\"RESULTS of {loops} Loops - Looping with Pandas Vectorization\")\n",
        "print(res)\n",
        "print(f\"Mean: {np.mean(num, axis=0)}, StdDev: {np.std(num, axis=0)}\")\n",
        "mean_res.append(np.mean(num, axis=0))\n",
        "std_res.append(np.std(num, axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WN1zB1bslcGC"
      },
      "source": [
        "### Vectorization to the Next Level!  Numpy arrays!\n",
        "\n",
        "The NumPy library, which describes itself as a “fundamental package for scientific computing in Python”, performs operations under the hood in optimized, pre-compiled C code. \n",
        "*  Like Pandas, NumPy operates on array objects (referred to as ndarrays)\n",
        "*  Numpy eliminates overhead inherent in Pandas series (such as indexing, data type checking, etc). \n",
        "\n",
        "As a result, operations on NumPy arrays can be significantly faster than operations on Pandas series.  See below for a dramatic example!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H9jMk6klRv4L"
      },
      "source": [
        "res = []\n",
        "num = []\n",
        "\n",
        "# Haversine applied on rows via iteration\n",
        "for l in range(0,loops):\n",
        "    start = time.time()\n",
        "    # Vectorized implementation of Haversine applied on Pandas series\n",
        "    df['distance'] = haversine(32.266, 110.985, df['latitude'].values, df['longitude'].values)\n",
        "    end = time.time()\n",
        "    delta = (end - start)*1000\n",
        "    msg = (\"%7.3F ms\" % delta)  \n",
        "    res.append(msg)\n",
        "    num.append(delta)\n",
        "name.append('Numpy Vectorization')\n",
        "print(f\"RESULTS of {loops} Loops - Looping with Numpy Vectorization\")\n",
        "print(res)\n",
        "print(f\"Mean: {np.mean(num, axis=0)}, StdDev: {np.std(num, axis=0)}\")\n",
        "mean_res.append(np.mean(num, axis=0))\n",
        "std_res.append(np.std(num, axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xi-iqNWUzk44"
      },
      "source": [
        "## Print Results of these Looping Techniques"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLmMcd2Gnl6w"
      },
      "source": [
        "results = pd.DataFrame()\n",
        "results['Type of Looping'] = name\n",
        "results['Mean Loop time (ms)'] = mean_res\n",
        "results['Loop StdDev (ms)'] = std_res\n",
        "results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zRYJZ1EkI4Nr"
      },
      "source": [
        "# Optimizing Data Ingest\n",
        "## Data Generators\n",
        "\n",
        "### Why is this here?\n",
        "\n",
        "Two of the challenges that we can have with large datasets is that first, we may not have enough RAM available to load the entire dataset into memory.  This can be particularly challenging when we are using GPU's to train.  Second, when a data generator is NOT used, it is very difficult to use multiprocessing to parallelize the training of the model.  Data Generators, like Keras' fit_generator method (now its just model.fit()), allow us to handle both of these challenges.  Plus, as you'll see, when the file is too large to fit into memory, data generators are faster than chunking data using Pandas.\n",
        "\n",
        "Below we develop a somewhat complex data generator that allows one to filter data row by row.  The valuable thing comes in the timing numbers, however, as this approach is much faster than doing a Pandas read_csv() chunksize call.\n",
        "\n",
        "### What will we do?\n",
        "We will open our COVID JHU datafile and compare three methods of bringing the data to the GPU/CPU(s) doing the training.\n",
        "1.  Our example data generator (which is a bit more complex than we need here, but it allows us to specify criteria for grabbing data)\n",
        "2.  A standard Pandas 100% in-memory approach\n",
        "3.  An optimization of the Pandas 100% in-memory approach\n",
        "4.  A Pandas data chunking approach (helps if you don't have enough memory)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iZR1_VX0aaVm"
      },
      "source": [
        "#  First lets restart the runtime to reset memories\n",
        "\n",
        "import os\n",
        "\n",
        "def restart_runtime():\n",
        "  os.kill(os.getpid(), 9)\n",
        "\n",
        "restart_runtime()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lo7Zin1LueBT"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import csv\n",
        "import time\n",
        "import resource\n",
        "import os\n",
        "import psutil\n",
        "\n",
        "# Download a reasonably large datafile.  Right now we're choosing JHU COVID Confirmed Case data\n",
        "filename = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_US.csv'\n",
        "!wget -O /content/datafile 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_US.csv'\n",
        "\n",
        "df = pd.read_csv('/content/datafile')\n",
        "\n",
        "print(df.info())\n",
        "\n",
        "def reduce_mem_usage(df, verbose=True):\n",
        "    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n",
        "    start_mem = df.memory_usage().sum() / 1024**2\n",
        "\n",
        "    # Walk through each feature in our DataFrame and determine if we can make use of a smaller\n",
        "    # datatype    \n",
        "    for col in df.columns:\n",
        "        col_type = df[col].dtypes\n",
        "        if col_type in numerics:\n",
        "            c_min = df[col].min()\n",
        "            c_max = df[col].max()\n",
        "\n",
        "            # Process Integer Datatypes to pick the optimal type for memory optimization\n",
        "            if str(col_type)[:3] == 'int':\n",
        "                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n",
        "                    df[col] = df[col].astype(np.int8)\n",
        "                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n",
        "                    df[col] = df[col].astype(np.int16)\n",
        "                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n",
        "                    df[col] = df[col].astype(np.int32)\n",
        "                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n",
        "                    df[col] = df[col].astype(np.int64)  \n",
        "            else:\n",
        "                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n",
        "                    df[col] = df[col].astype(np.float16)\n",
        "                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n",
        "                    df[col] = df[col].astype(np.float32)\n",
        "                else:\n",
        "                    df[col] = df[col].astype(np.float64)    \n",
        "    end_mem = df.memory_usage().sum() / 1024**2\n",
        "    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n",
        "    return df\n",
        "\n",
        "def get_mem_size():\n",
        "    process = psutil.Process(os.getpid())\n",
        "    return (process.memory_info().rss)  # in bytes \n",
        "\n",
        "def getstuff(filename, criterion):\n",
        "    with open(filename, \"r\") as csvfile:\n",
        "        datareader = csv.reader(csvfile)\n",
        "        yield next(datareader)  # yield the header row\n",
        "        count = 0\n",
        "        for row in datareader:\n",
        "            #print(row)\n",
        "            '''if row[3] == criterion:\n",
        "                yield row\n",
        "                count += 1\n",
        "            elif count:\n",
        "                # done when having read a consecutive series of rows \n",
        "                return'''\n",
        "            yield(row)\n",
        "\n",
        "# This is our data generator\n",
        "\n",
        "def getdata(filename, criteria):\n",
        "    for criterion in criteria:\n",
        "        for row in getstuff(filename, criterion):\n",
        "            yield row\n",
        "sequence_of_criteria = [109.379, 98.664] \n",
        "\n",
        "starting_memory = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\n",
        "sm = (get_mem_size())\n",
        "\n",
        "start = time.time()\n",
        "\n",
        "# Compare timing between the data generator and two types of Pandas call\n",
        "#\n",
        "time_arr = []\n",
        "mem_arr = []\n",
        "# ------------------------------------------------------------------------------\n",
        "#\n",
        "# Data Generator Method\n",
        "#\n",
        "print('DATA GENERATOR OUTPUT BELOW\\n')\n",
        "for row in getdata('/content/datafile', sequence_of_criteria):\n",
        "    continue\n",
        "end = time.time()\n",
        "msg = (\"Generator Time: %7.3F ms\" % ((end - start)*1000))\n",
        "time_arr.append(msg)\n",
        "\n",
        "# The below will be in KB on Linux (CoLab)\n",
        "generator_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - starting_memory\n",
        "print('Memory usage should be roughly zero.  A generator is lazy: instead of creating all of the strings immediately, it will generate one string at a time on demand')\n",
        "print(\"MEMORY USED: %d Kb\\n\" % generator_mem )\n",
        "mem_arr.append(generator_mem)\n",
        "dgmem = get_mem_size() - sm\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YPGKQn85KAfT"
      },
      "source": [
        "#\n",
        "# Pandas 100% in memory\n",
        "#\n",
        "print('PANDAS UNOPTIMIZED 100% IN MEMORY OUTPUT BELOW\\n')\n",
        "starting_memory = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\n",
        "start = time.time()\n",
        "%time df = pd.read_csv('/content/datafile')\n",
        "\n",
        "end = time.time()\n",
        "msg = (\"Pandas ReadCSV Time: %7.3F ms\" % ((end - start)*1000) )  \n",
        "time_arr.append(msg)\n",
        "pandas_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - starting_memory\n",
        "print('Memory Usage will be large because Pandas allocates the whole file plus lots of overhead to memory')\n",
        "print(\"MEMORY USED: %d Kb\\n\" % pandas_mem )\n",
        "mem_arr.append(pandas_mem)\n",
        "print('df describe below so you can see the unoptimized datatypes\\n')\n",
        "print(df.dtypes[0:40])\n",
        "\n",
        "print(\"Pandas Memory Usage:\")\n",
        "df.memory_usage().sum()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hXHz6j_ZKR6o"
      },
      "source": [
        "#\n",
        "# Pandas 100% in memory - We will optimize datatypes in read_csv\n",
        "#\n",
        "print('PANDAS OPTIMIZED 100% IN MEMORY OUTPUT BELOW\\n')\n",
        "\n",
        "# Optimize memory usage from the dataframe\n",
        "df = reduce_mem_usage(df)\n",
        "\n",
        "dtypes = df.dtypes\n",
        "colnames = dtypes.index\n",
        "types = [i.name for i in dtypes.values]\n",
        "column_types = dict(zip(colnames, types))\n",
        "\n",
        "start = time.time()\n",
        "starting_memory = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\n",
        "# Import the dataframe again, but this time with optimized datatypes\n",
        "%time df = pd.read_csv('/content/datafile', dtype=column_types)\n",
        "\n",
        "end = time.time()\n",
        "msg = (\"Pandas ReadCSV (optimized) Time: %7.3F ms\" % ((end - start)*1000) )  \n",
        "time_arr.append(msg)\n",
        "pandas_opt_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - starting_memory\n",
        "mem_arr.append(pandas_opt_mem)\n",
        "print('Memory Usage Should be Smaller than the Previous')\n",
        "print(\"MEMORY USED: %d Kb\\n\" % pandas_opt_mem )\n",
        "print('df datatypes after memory reduction')\n",
        "print(df.dtypes[0:40])\n",
        "\n",
        "print(\"Pandas Memory Usage:\")\n",
        "df.memory_usage().sum()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "azSepA1cKVKn"
      },
      "source": [
        "#\n",
        "# Pandas Data Chunking approach\n",
        "#\n",
        "print('PANDAS CHUNKSIZE = 50 records OUTPUT BELOW\\n')\n",
        "start = time.time()\n",
        "starting_memory = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\n",
        "\n",
        "# Set a chunk size of 50 records\n",
        "for df in pd.read_csv('/content/datafile', chunksize=50):\n",
        "    continue\n",
        "print(df.info())\n",
        "end = time.time()\n",
        "msg = (\"Pandas ReadCSV Chunking Time:     %7.3F ms\" % ((end - start)*1000) )\n",
        "time_arr.append(msg)\n",
        "mem_one_chunk = (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - starting_memory)\n",
        "mem_arr.append(mem_one_chunk)\n",
        "print('Memory usage will be smaller due to chunksize of 50')\n",
        "print(\"MEMORY USED per CHUNK: %d KBytes\\n\" % (mem_one_chunk))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oAFbt1ByNSnj"
      },
      "source": [
        "##  Print out array that captures the timing of each method:\n",
        "* Data Generator\n",
        "* Pandas read_csv 100% in memory\n",
        "* Optimized read_csv in memory aproach\n",
        "* Pandas read_csv chunking approach\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lS_jrXVbUVbo"
      },
      "source": [
        "print (time_arr)\n",
        "print (mem_arr)\n",
        "results2 = pd.DataFrame()\n",
        "results2['Approach and Time for Data Ingest'] = time_arr\n",
        "results2['Memory Consumed (approx. Kbytes)'] = mem_arr\n",
        "results2\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oj0fuwMrdkNs"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}