Lab 1 - MAT 128 review.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EJ9zjSSvVrT0"
   },
   "source": [
    "# Lab 1: MAT 128 Review\n",
    "\n",
    "This lab reviews the most important data manipulation, filtering, and plotting techniques from MAT 128:\n",
    "\n",
    "* data frames \n",
    "* loading CSV files, including date columns\n",
    "* missing data\n",
    "* mean, median, and standard deviation of a column\n",
    "* line plots\n",
    "* accessing date information\n",
    "* histogram\n",
    "* creating new columns from existing ones\n",
    "* bar charts\n",
    "* filtering\n",
    "\n",
    "We explore the New York City weather data from 2014 and 2015.  This data was curated by [FiveThirtyEight.com](fivethirtyeight.com) and used in their article [What 12 Months of Record-Setting Temperatures Looks Like Across the U.S.](https://fivethirtyeight.com/features/what-12-months-of-record-setting-temperatures-looks-like-across-the-u-s/).\n",
    "\n",
    "### Section 1.1: Loading the data\n",
    "\n",
    "First import the necessary *libraries* (also called *modules* or *packages*):\n",
    "* [pandas](https://pandas.pydata.org/): data analysis tools\n",
    "* [matplotlib](https://matplotlib.org/): plotting\n",
    "* [seaborn](https://seaborn.pydata.org/):  higher-level data visualization built on the MatPlotLib library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "e8iGcghKc4wn"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "# The following line is needed for some versions of Python and Jupyter Notebooks to display the plots in the notebook.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SZRpGnt0dVs_"
   },
   "source": [
    "We will now read the data file, which is in *Comma-Separated Values (CSV)* format, into a *DataFrame*. \n",
    "\n",
    "Recall that a CSV file is a text file that stores the rows of a table (ex. an Excel table) by separating data values in different columns with commas. \n",
    "\n",
    "A *DataFrame* is how the pandas library (and hence Python) stores a table of data in the computer.\n",
    "\n",
    "We can load the CSV file from a URL into the DataFrame `weather`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "F4tL7D50jrdu"
   },
   "outputs": [],
   "source": [
    "weather = pd.read_csv(\"https://raw.githubusercontent.com/fivethirtyeight/data/master/us-weather-history/KNYC.csv\")\n",
    "weather"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "krKgdLdD7jLT"
   },
   "source": [
    "Or if you have the file in the same directly as this notebook (hard to do on Colab), you can load the CSV file into the DataFrame `weather` as we did in MAT 128:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "wIgg0ulm7iJz"
   },
   "outputs": [],
   "source": [
    "weather = pd.read_csv(\"KNYC.csv\")\n",
    "weather"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HYr57h5GkWl8"
   },
   "source": [
    "Look at the DataFrame to answer the following questions:\n",
    "* When does the data begin?  \n",
    "* When does it end?   \n",
    "* What kinds of information do we have about the weather?\n",
    "\n",
    "In the data column names, \"actual\" refers to the temperature or precipitation (in inches) value recorded on that date, \"average\" is the historical average value for that day, and \"record\" is the highest or lowest recorded value for that day.\n",
    "\n",
    "To only display the first 5 rows of the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "XoCMO2YmkIps"
   },
   "outputs": [],
   "source": [
    "weather.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ncn72ZHfVLgb"
   },
   "source": [
    "### Section 1.2 Dates\n",
    "While we can see that the `date` column is a date, it is not explicitly stored in the DataFrame as a data, and thus we can't use commands for dates on it.\n",
    "\n",
    "The following code shows the *data type* of each column, which is how the data is stored in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "EGPRylDwlr00"
   },
   "outputs": [],
   "source": [
    "weather.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TM0IWPt8lwK5"
   },
   "source": [
    "The following code tells Python to encode the `date` column as a DateTime object, and then displays the updated DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "RkjG4qfnVLgc"
   },
   "outputs": [],
   "source": [
    "weather[\"date\"] = pd.to_datetime(weather[\"date\"])\n",
    "weather.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u2dHPyssmDNx"
   },
   "source": [
    "Do you notice any difference in the `date` column compared to the original DataFrame?\n",
    "\n",
    "Display the data types of each column again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "bQZonLtFVLgc"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yKkUBRhqmZMz"
   },
   "source": [
    "How have the data types changed?\n",
    "\n",
    "### Section 1.3 Missing data and column statistics\n",
    "\n",
    "It looks like we have a year's worth of weather data from July 1, 2014 to June 30, 2015.  See if there is any missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "jyJK94CUVLge"
   },
   "outputs": [],
   "source": [
    "weather.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j8EaJ5OHVLge"
   },
   "source": [
    "The `count` row tells us how many values are in each column.  Since the count is 365 for each column, and this is the number of rows in our DataFrame, then we know there is no missing data.\n",
    "\n",
    "The `describe()` function also displays the 5-number summary (minimum data value, 25 percentile, 50 percentile or *median*, 75 percentile, and maximum data value) for each numerical column, as well as the mean and standard deviation.\n",
    "\n",
    "What is the standard deviation of the average minimum temperature?\n",
    "\n",
    "What is the median actual precipitation?\n",
    "\n",
    "What is the 75th percentile of record maximum temperature?\n",
    "\n",
    "### Section 1.4 Line Plots\n",
    "\n",
    "Recall that a line plot shows how some data value changes over time.  The following code creates a line plot of the daily mean temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "0BRm9_tIVLgf"
   },
   "outputs": [],
   "source": [
    "weather.plot(x = \"date\", y= \"actual_mean_temp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TNTPW8FjVLgf"
   },
   "source": [
    "What trends do you notice in the above plot?  Does this make sense?\n",
    "\n",
    "Remember, our plots should always have a title and axis labels.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "nBmIctZWVLgg"
   },
   "outputs": [],
   "source": [
    "weather.plot(x = \"date\", y= \"actual_mean_temp\",legend = False)\n",
    "plt.title(\"Daily mean temperature\")\n",
    "plt.xlabel(\"Date\")\n",
    "plt.ylabel('Mean temperature (F)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_1YpVx-Es1z2"
   },
   "source": [
    "Make a line plot of the daily precipitation, including axes labels and a title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "SsKivj5Hs6yj"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TMqaSTSxtDeF"
   },
   "source": [
    "What trends do you notice in your plot?  How does this compare with the plot of the mean daily temperature? \n",
    "\n",
    "Do you think a line plot is the best way to understand the daily precipiation?\n",
    "\n",
    "### Section 1.5 Histograms\n",
    "\n",
    "Let's look at another way to visualize the daily precipitation data: plotting the distribution of the daily precipitation.  Recall that since the precipitation data is quantitative, we do this with a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "zZqURv9FVLgh"
   },
   "outputs": [],
   "source": [
    "weather[\"actual_precipitation\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1TlymKTZVLgh"
   },
   "source": [
    "Sometimes it is helpful to have a finer breakdown of histogram bins.  Do this by adding the *parameter* `bins = 20` in between the parentheses of the function `hist()` like this: \n",
    "`weather[\"actual_precipitation\"].hist(bins = 20)`\n",
    "\n",
    "A parameter gives a function additional information.\n",
    "\n",
    "Add a title and axes labels to your plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "zxvPDpZxVLgh"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zvBygHTouVW7"
   },
   "source": [
    "What do you notice about the daily precipitation distribution?  Is this an approximately normal distribution?  Why or why not?\n",
    "\n",
    "### Section 1.6 Creating new columns and bar charts\n",
    "\n",
    "Maybe it would be helpful to make a bar chart of the number of days with 0 precipitation and the number of days with >0 precipitation.  To do this, we will first make a new column in our DataFrame that contains a 1 if there was precipitation that day, and a 0 if there was not.\n",
    "\n",
    "Define a function that outputs (returns) 1 if the input value is greater than 0, and 0 if it is not.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "UGF_fD9TvwCO"
   },
   "outputs": [],
   "source": [
    "def is_positive(x):\n",
    "  if x > 0:\n",
    "    return 1\n",
    "  else:\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OXZRhOR2v4vM"
   },
   "source": [
    "Now we use the `apply()` function to pass the value from each row of the `actual_precipitation` column into our `is_positive()` function, and store the output for that row in a new column called `precipitated`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "U2lqtQcowZd7"
   },
   "outputs": [],
   "source": [
    "weather[\"precipitated\"] = weather[\"actual_precipitation\"].apply(is_positive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_tAC0o_MwsaX"
   },
   "source": [
    "Display the `weather` DataFrame again to see what happened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "MutkBlcSwyqT"
   },
   "outputs": [],
   "source": [
    "weather.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4NE5sWsfw3yi"
   },
   "source": [
    "Is this what you expected?\n",
    "\n",
    "Now let's make a bar chart of the `precipitated` column.  Remember we must compute the value counts first when making a bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "-erUW6ddw_sr"
   },
   "outputs": [],
   "source": [
    "precip_counts = weather[\"precipitated\"].value_counts()\n",
    "precip_counts.plot(kind = \"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SvUWLSkixMLN"
   },
   "source": [
    "Were there more days with or without precipitation?  Is this what you would expect?\n",
    "\n",
    "Add axes labels and a title to the bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "R6nOHGc0xXL3"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "REkTkubCxizu"
   },
   "source": [
    "Instead of precipitation, we might be interested in the number of hot (or cold) days.  Create a bar chart showing the number of days with a max temperature greater than 80F and the number of days with a max temperature of 80F or less.  \n",
    "\n",
    "(Alternatively, choose your own temperature cut-off.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "p-_kOPTuyPDw"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "-9nRLWptyO1B"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JUPhKyVzVLgi"
   },
   "source": [
    "### Section 1.7 Filtering\n",
    "\n",
    "What about if we only want to look at part of our data, such as the month of February?  Then we need to *filter* our dataset.\n",
    "\n",
    "Let's plot the distribution of daily minimum temperatures in February.\n",
    "\n",
    "First we need to filter our data to only includes rows from February.  Recall that we can access just the month part of the date by adding `.dt.month` to the `date` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "t8ZlNyglVLgk",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "feb_filter = weather[\"date\"].dt.month == 2\n",
    "weather[feb_filter]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oGwI7kTKqxi9"
   },
   "source": [
    "Store this new DataFrame with only the February rows in a new variable, and then create a histogram of the distribution of the minimum daily temperatures in February 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "ghrmNP_nVLgk"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "02Q6CN4TrdMv"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k3hkHSI8VLgk"
   },
   "source": [
    "See https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#time-date-components for the different properties that can be used after dt.  For example to get the mean of the historical average precipitation for all Tuesdays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "eMpsZT0lVLgl"
   },
   "outputs": [],
   "source": [
    "tues_filter = weather[\"date\"].dt.dayofweek == 1\n",
    "tues_weather = weather[tues_filter]\n",
    "tues_weather[\"average_precipitation\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tjxVRMHYr2vN"
   },
   "source": [
    "Can you compute the standard deviation (`.std()`) of the record maximum temperature in the first quarter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "AqviUMcdsLQm"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4PFpfdOP7qj1"
   },
   "source": [
    "Reference for functions seen in MAT 128 that compute measures of center and spread of a column:\n",
    "* `.mean()` - computes the mean\n",
    "* `.median()` - computes the median\n",
    "* `.mode()` - computes the mode \n",
    "* `.var()` - computes the variance\n",
    "* `.std()` - computes the standard deviation\n",
    "* `.min()` - finds the minimum value in the column\n",
    "* `.max()` - finds the maximum value in the column\n",
    "* `.idxmin()` - find the row number containing the minimum value in the column\n",
    "* `.idxmax()` - find the row number containing the maximum value in the column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X90CgwVMsME4"
   },
   "source": [
    "#### Optional challenge questions:\n",
    "* Create a bar chart of the number of record maximum temperatures that occured in the year 2000 or later and the number that occured before 2000\n",
    "* There is an English proverb that \"March comes in like a lion and out like a lamb\".  Is the mean temperature (or precipitation) at the beginning of March significantly lower (or higher) than at the end of March?\n",
    "* What would you like to know about the NYC weather in this data set?  Plot or compute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "NRNhl-uMtaDv"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Lab 1 - MAT 128 review.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}