Lab 12 - Comparing distributions visually.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 12 - Comparing distributions visually \n",
    "\n",
    "This lab will use two days worth of data from the 311 Service Request dataset on NYC Open Data. \n",
    "\n",
    "### 12.1 Download and read in the data\n",
    "\n",
    "We will download the 311 data from Sunday October 6 and Monday October 7, 2024:\n",
    "1. Go to https://nycopendata.socrata.com/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9 and click \"Actions\" and then \"Query Data\" in the menu. \n",
    "2. Filter the data to only contain complaints made on October 6 and 7, 2024: \n",
    "\n",
    "    a. Below the data, click on the drop down box \"Select a column to filter...\" and select \"Created Date\".\n",
    "    \n",
    "    b. Change the first date to be \"2024 Oct 06 12:00:00 AM\"\n",
    "    \n",
    "    c. Change the second date to be \"2024 Oct 07 11:59:59 PM:\n",
    "    \n",
    "    d. Click the blue \"Apply\" button at the bottom of the page.\n",
    "    \n",
    "    e. To download the filtered data, click the \"Export\" button above the data on the right, and then click the \"Download\" button in the pop-up window.\n",
    "    \n",
    "3. If necessary, rename the file so that it does not have the same name as the previous 311 data file.\n",
    "4. Upload your new data file to Jupyter Hub.\n",
    "\n",
    "As usual we import the matplotlib and pandas packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load your new 311 data file into the dataframe `calls`.  Use the `parse_dates` parameter to store the `Created Date` column as a `DatetimeIndex` type instead of string (see Lab 2 if you forget how to do this).  Recall that this parameter tells Pandas that the `Created Date` column in actually a date/time and not just some random text.  Parsing the date takes some extra time, so we have been skipping this step if we are not using the date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "    <code>df_name = pd.read_csv(\"data_file.csv\",parse_dates=[\"column_to_read_as_date\"])</code>\n",
    "</details>\n",
    "\n",
    "Display `calls` to check that it was created correctly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12.2 Comparing complaint times on Sunday vs. Monday\n",
    "\n",
    "First we are going to compare the times complaints were made on Sunday October 1 with the times complaints were made on Monday October 2.  We will use histograms plotted on top of each other to visualize and compare the two distributions.  Before starting, how do you think timing of complaints will differ between Sunday and Monday?\n",
    "\n",
    "First we will create a filter to find all complaints created on October 1.  Type `sun_filter = calls[\"Created Date\"].dt.day ==1` below and run the code.  The `.dt.day` gets just the day from the `DatetimeIndex` type and `==1` checks if it is equal to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use your filter `sun_filter` to create a new dataframe called `sun_calls` containing just the complaints made on Sunday.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "    <code>new_df_name = df[filter]</code>\n",
    "</details>\n",
    "\n",
    "Now create a dataframe called `mon_calls` containing only the calls from Monday October 2.  That is, create the filter and then use it to select from the `calls` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now plot a histogram of the hours of the Sunday calls.  We can get a Series of the hour each Sunday call was created with `sun_calls[\"Created Date\"].dt.hour`.  Type and run this below to check this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So to make a histogram of the hours, type `sun_calls[\"Created Date\"].dt.hour.hist()` below and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the distribution?  You may want to increase the number of bins to 24.\n",
    "\n",
    "To plot two histograms on the same graph, put the code to plot one histogram on the first line of a cell, and the code to plot the other histogram on the second line of the same cell.  \n",
    "\n",
    "In the cell below, replot the histogram of the call hours for the Sunday complaints, along with a histogram of the call hours of the Monday complaints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first histogram (Sunday) is in blue and the second (Monday) in orange.  Can you see all of both histograms?  What do you notice?\n",
    "\n",
    "### 12.3 Improving Histogram Appearance\n",
    "\n",
    "Let's make this plot look nicer.  To make a histogram transparent, add the parameter `alpha = 0.5`.  You can also change the color of a histogram to red by adding the parameter `color = \"red\"`.  A list of the possible colors is [here](https://matplotlib.org/examples/color/named_colors.html).  \n",
    "\n",
    "Can you add axis labels and a title?  \n",
    "\n",
    "Finally, we can add a legend with `plt.legend([\"Sunday\",\"Monday\"])` (This legend assumes the first histogram is for Sunday and the second is for Monday.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?  What differences are there between the distributions of the times of the 311 complaints on Sunday versus Monday?\n",
    "\n",
    "There may be more complaints on one day than on the other.  So instead of comparing the absolute number of complaints at each time, it might be better to compare the proportion of complaints at each time.  Do this by adding the parameter `density = True` to each histogram function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12.4 Comparing complaint distribution by borough on Monday vs. Sunday\n",
    "\n",
    "Now we will compare the distribution of complaints by borough on Sunday versus Monday.  Do you think the distribution will change?  If so, how?\n",
    "\n",
    "Above we created `sun_calls` as a dataframe of only the Sunday calls and `mon_calls` as a dataframe of only the Monday calls.\n",
    "\n",
    "First, get the value counts of the `Borough` column in `sun_calls` and in `mon_calls` and store these in the variables `sun_borough_counts` and `mon_borough_counts` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "    <code>counts_var = df[\"colunm_name\"].value_counts()</code>\n",
    "</details>\n",
    "\n",
    "To combine two bar plots into a single plot, we need to combine the two value counts into a new dataframe.  The code below creates a new dataframe called `borough_counts_df` with the column containing the Sundays value counts called `sunday` and the column containing the Monday value counts called `monday`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "borough_counts_df = pd.DataFrame({\"sunday\":sun_borough_counts,\"monday\":mon_borough_counts})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the new dataframe `borough_counts_df` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make a combined bar chart of these two columns, type and run the code: `borough_counts_df.plot(kind = \"bar\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How did the distribution of 311 service requests across boroughs change from Sunday to Monday?\n",
    "\n",
    "### 12.5 Improving bar plot appearance\n",
    "\n",
    "We can improve the appearance of this bar chart in a few ways.  To change the color of the bars to purple and aqua (or whatever colors you prefer), add the parameter `color = [\"purple\",\"turquoise\"]`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>borough_counts_df.plot(kind = \"bar\", color = [\"purple\",\"turquoise\"])</code>\n",
    "</details>\n",
    "\n",
    "Next try adding axis labels and a title.  You can also change the wording of the current legend by adding a new one like we did for the histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to display the proportion of 311 calls from each borough instead of the number of calls, we need to *normalize* the counts first by dividing each value count by the total number of counts for that day.  We do this by adding the parameter `normalize = True` when computing the value counts originally.\n",
    "\n",
    "Run the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "norm_sun_borough_counts = sun_calls[\"Borough\"].value_counts(normalize = True)\n",
    "norm_sun_borough_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the ranking of boroughs compare to the ranking for the original value counts?\n",
    "\n",
    "Normalize the Monday borough counts, storing them in the variable `norm_mon_borough_counts`, and displaying them to check your code worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>norm_mon_borough_counts = mon_calls[\"Borough\"].value_counts(normalize = True)</code>\n",
    "</details>\n",
    "\n",
    "How does the order of boroughs for these normalized counts compare to the original borough order? \n",
    "\n",
    "Next, combine the two normalized value counts into a new dataframe called `norm_borough_counts_df`, and display it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>norm_borough_counts_df = pd.DataFrame({\"sunday\":norm_sun_borough_counts,\"monday\":norm_mon_borough_counts})\n",
    "        \n",
    "norm_borough_counts_df</code>\n",
    "</details>\n",
    "\n",
    "Finally, plot the normalized data in a combined bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Hint:</summary>\n",
    "See the end of section 12.4.\n",
    "</details>\n",
    "\n",
    "How is the bar chart of the normalized data similar to the bar chart of the counts?  How are they different?\n",
    "\n",
    "### 12.6 Comparing daytime and night-time calls\n",
    "We can make all sorts of interesting comparisons with filters.  Here is one comparing the type of calls made at night (6pm - 6am) with the type of calls made during the day.  Try running the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "before_6am_filter = calls[\"Created Date\"].dt.hour < 6\n",
    "after_6pm_filter = calls[\"Created Date\"].dt.hour > 18\n",
    "\n",
    "night_calls = calls[before_6am_filter | after_6pm_filter]\n",
    "day_calls = calls[~before_6am_filter & ~after_6pm_filter]\n",
    "\n",
    "night_counts = night_calls[\"Complaint Type\"].value_counts()\n",
    "day_counts = day_calls[\"Complaint Type\"].value_counts()\n",
    "night_day_df = pd.DataFrame({\"Day\":day_counts, \"Night\": night_counts})\n",
    "\n",
    "# Plot only the complaints that occur more than 100 times during the day \n",
    "# and more than 50 times during the night.\n",
    "day_top_filter = night_day_df[\"Day\"] > 100\n",
    "night_top_filter =  night_day_df[\"Night\"] >50\n",
    "night_day_df[day_top_filter & night_top_filter].plot(kind = \"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Challenges\n",
    "- Plot overlapping histograms of the hours that residential noise and commerical noise complaints are made.  Residential noise complaints are listed as `Noise - Residential` under `Complaint Type` and commerical noise complaints are listed as `Noise - Commercial` under `Complaint Type`.\n",
    "- Plot overlapping bar charts of the borough distribution of no heat/hot water complaints (`HEAT/HOT WATER`) and another complaint of your choice.  Is there a different in the distributions?\n",
    "- Your choice!  Pick one variable to compare in two different situations.  You can find a list of the different types of information you can get from the`DatetimeIndex` [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DatetimeIndex.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}