data_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Data-Analysis-with-CSVs\" data-toc-modified-id=\"Data-Analysis-with-CSVs-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Data Analysis with CSVs</a></span></li><li><span><a href=\"#Objectives\" data-toc-modified-id=\"Objectives-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Objectives</a></span></li><li><span><a href=\"#Loading-a-CSV\" data-toc-modified-id=\"Loading-a-CSV-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Loading a CSV</a></span><ul class=\"toc-item\"><li><span><a href=\"#Get-the-File-Path\" data-toc-modified-id=\"Get-the-File-Path-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Get the File Path</a></span></li><li><span><a href=\"#Inspecting-CSV-files\" data-toc-modified-id=\"Inspecting-CSV-files-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Inspecting CSV files</a></span><ul class=\"toc-item\"><li><span><a href=\"#Printing-Lines-in-Python\" data-toc-modified-id=\"Printing-Lines-in-Python-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Printing Lines in Python</a></span></li><li><span><a href=\"#Printing-Lines-in-bash\" data-toc-modified-id=\"Printing-Lines-in-bash-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Printing Lines in bash</a></span></li><li><span><a href=\"#Discussion\" data-toc-modified-id=\"Discussion-3.2.3\"><span class=\"toc-item-num\">3.2.3&nbsp;&nbsp;</span>Discussion</a></span></li></ul></li><li><span><a href=\"#Using-the-csv-library\" data-toc-modified-id=\"Using-the-csv-library-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Using the <code>csv</code> library</a></span></li></ul></li><li><span><a href=\"#Data-Preparation\" data-toc-modified-id=\"Data-Preparation-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Data Preparation</a></span><ul class=\"toc-item\"><li><span><a href=\"#Clean-the-Sale-Prices\" data-toc-modified-id=\"Clean-the-Sale-Prices-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Clean the Sale Prices</a></span></li><li><span><a href=\"#Clean-the-Dates\" data-toc-modified-id=\"Clean-the-Dates-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Clean the Dates</a></span><ul class=\"toc-item\"><li><span><a href=\"#Using-Python-built-in-methods\" data-toc-modified-id=\"Using-Python-built-in-methods-4.2.1\"><span class=\"toc-item-num\">4.2.1&nbsp;&nbsp;</span>Using Python built-in methods</a></span></li></ul></li></ul></li><li><span><a href=\"#Data-Analysis\" data-toc-modified-id=\"Data-Analysis-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Data Analysis</a></span><ul class=\"toc-item\"><li><span><a href=\"#Question-1:-What-is-the-range-of-years-for-which-we-have-sales-data?\" data-toc-modified-id=\"Question-1:-What-is-the-range-of-years-for-which-we-have-sales-data?-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Question 1: What is the range of years for which we have sales data?</a></span></li><li><span><a href=\"#Question-2:-How-many-properties-sold-in-2020?\" data-toc-modified-id=\"Question-2:-How-many-properties-sold-in-2020?-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Question 2: How many properties sold in 2020?</a></span></li><li><span><a href=\"#Question-3:-What-was-the-mean-sale-price-in-2020?\" data-toc-modified-id=\"Question-3:-What-was-the-mean-sale-price-in-2020?-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Question 3: What was the mean sale price in 2020?</a></span></li><li><span><a href=\"#Question-4:-How-many-properties-sold-in-each-year?\" data-toc-modified-id=\"Question-4:-How-many-properties-sold-in-each-year?-5.4\"><span class=\"toc-item-num\">5.4&nbsp;&nbsp;</span>Question 4: How many properties sold in each year?</a></span></li></ul></li><li><span><a href=\"#Chart-the-Data\" data-toc-modified-id=\"Chart-the-Data-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Chart the Data</a></span></li><li><span><a href=\"#Level-Up:-Counter\" data-toc-modified-id=\"Level-Up:-Counter-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Level Up: <code>Counter</code></a></span></li><li><span><a href=\"#Level-Up:-Using-datetime\" data-toc-modified-id=\"Level-Up:-Using-datetime-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Level Up: Using <code>datetime</code></a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis with CSVs\n",
    "\n",
    "In this lesson, we'll complete a full data analysis with a raw data file. Comma Separated Values files, known as CSVs, are one of the most common file formats for storing tabular data. In this lesson, we'll show you how to load one into memory and work with it using Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives\n",
    "\n",
    "Perform a complete data analysis by...\n",
    "\n",
    "- Creating a Python data structure from a .csv file\n",
    "- Exploring and cleaning the data \n",
    "- Conducting descriptive analysis\n",
    "- Visualizing the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading a CSV\n",
    "\n",
    "The before you can work with a CSV file, you need to load it into memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the File Path\n",
    "\n",
    "Make sure you have the path to your data file. For this example, it is in the root folder of this repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This will only work on Mac, Linux, or a separate bash terminal in Windows\n",
    "\n",
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `.` to indicate the root folder of the repository when working within this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file_path = \"./Kings_County_Real_Estate_Sales.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting CSV files\n",
    "\n",
    "In a CSV file, each line represents one row of tabular data, and consecutive values in that row are separated by a comma. Often, the first row contains the column names separated by commas, also known as field names. Let's confirm that this is the case and learn about our dataset by printing the first two lines of the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing Lines in Python\n",
    "\n",
    "We use the `with open()` syntax to easily open and read the file in the notebook. Using this syntax will automatically close the file once the statement is done running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code prints the first line of the CSV file\n",
    "\n",
    "with open(csv_file_path) as csvfile:\n",
    "    print(csvfile.readline())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can print the second line to look at an example of one observation from our dataset. This represents one real estate sale in Kings County."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code prints the second line of the CSV file\n",
    "\n",
    "with open(csv_file_path) as csvfile:\n",
    "    csvfile.readline()\n",
    "    print(csvfile.readline())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing Lines in bash\n",
    "\n",
    "Alternatively, you could use a bash command to print the first two lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This will only work on Mac, Linux, or a separate bash terminal in Windows\n",
    "\n",
    "!head -n 2 ./Kings_County_Real_Estate_Sales.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "\n",
    "Based on these two lines, what can we infer about the contents of the data file? What questions arise for you about the data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the `csv` library\n",
    "\n",
    "The [`csv` library](https://docs.python.org/3/library/csv.html) lets us easily process data in CSV files. We will use it to read each row in the `.csv` file and store its information in a Python object we can use for further analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [`csv.DictReader`](https://docs.python.org/3/library/csv.html#csv.DictReader) object is a file reader, reading each row and then converting it to a special kind of dictionary called an [OrderedDict](https://docs.python.org/3/library/collections.html#collections.OrderedDict). By default, it turns each row into a dictionary, using the field names from the first row as the keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print OrderedDict from first row of CSV file \n",
    "\n",
    "with open(csv_file_path) as csvfile:\n",
    "    reader = csv.DictReader(csvfile)\n",
    "    print(next(reader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get all of the data out of our file and into dictionaries, and store those dictionaries in a new list called `sales`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = []\n",
    "\n",
    "with open(csv_file_path) as csvfile:\n",
    "    reader = csv.DictReader(csvfile)\n",
    "    for row in reader:\n",
    "        sales.append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation\n",
    "\n",
    "Now that we've gotten all of our data into a Python object, we can prepare it for analysis. Let's look at a sample observation and consider how we might want to process it to make it easier to analyze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** How might you clean these data to tmake them easier to analyze?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the Sale Prices\n",
    "\n",
    "First, let's convert the sale prices from strings to numbers, so we can do arithmetic with them. For example, we could calculate the average price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sale in sales:\n",
    "    # Save the document date to a variable\n",
    "    string_price = sale[\"SalePrice\"]\n",
    "    \n",
    "    # Cast the sale price to int and add it to each dictionary element of sales\n",
    "    sale[\"sale_price\"] = int(string_price)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the Dates\n",
    "\n",
    "Next we'll clean the dates so that we can easily get the month and year of each sale. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Python built-in methods\n",
    "\n",
    "**Activity**: Process the `sales` list to add numeric values for month and year to each entry.\n",
    "\n",
    "<details>\n",
    "    <summary><b><u>Click Here for Answer Code</u></b></summary>\n",
    "    \n",
    "    for sale in sales:\n",
    "        # Save the document date to a variable\n",
    "        string_date = sale[\"DocumentDate\"]\n",
    "        \n",
    "        # Extract the sale year and month from the string, and cast to int\n",
    "        sale_year = int(string_date[6:10])\n",
    "        sale_month = int(string_date[0:2])    \n",
    "    \n",
    "        # Add the sale year and month to each dictionary element of sales\n",
    "        sale[\"sale_year\"] = sale_year\n",
    "        sale[\"sale_month\"] = sale_month\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your work here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis\n",
    "\n",
    "Now that you have a cleaned list with the property sales data, we can finally start to perform some data analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1: What is the range of years for which we have sales data?\n",
    "\n",
    "<details>\n",
    "            <summary><b><u>Click Here for Answer Code</u></b></summary>\n",
    "    \n",
    "    sale_years = [sale['sale_year'] for sale in sales]\n",
    "    print(min(sale_years))\n",
    "    print(max(sale_years))\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your work here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2: How many properties sold in 2020?\n",
    "\n",
    "<details>\n",
    "            <summary><b><u>Click Here for Answer Code</u></b></summary>\n",
    "    \n",
    "    \n",
    "    sale_years = [sale['sale_year'] for sale in sales]\n",
    "    sale_years_2020 = [year for year in sale_years if year == 2020]\n",
    "    len(sale_years_2020)\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your work here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3: What was the mean sale price in 2020?\n",
    "\n",
    "<details>\n",
    "            <summary><b><u>Click Here for Answer Code</u></b></summary>\n",
    "    \n",
    "    sale_prices_2020 = [sale['sale_price'] for sale in sales if sale['sale_year'] == 2020]\n",
    "    sum(sale_prices_2020)/len(sale_prices_2020)\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your work here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4: How many properties sold in each year?\n",
    "\n",
    "<details>\n",
    "            <summary><b><u>Click Here for Answer Code</u></b></summary>\n",
    "   \n",
    "    sale_year_count = []\n",
    "    sale_years = [sale['sale_year'] for sale in sales]\n",
    "    unique_sale_years = set(sale_years)\n",
    "    for unique_year in unique_sale_years:\n",
    "        num_sales = len([year for year in sale_years if year == unique_year])\n",
    "        sale_year_count.append((unique_year, num_sales))\n",
    "        print(unique_year, num_sales)\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your work here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chart the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `list` of `tuples` is helpful, but it takes a lot of effort to read it and make sense of the trends - let's visualize the results to make this easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a figure and one plot\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10, 5))\n",
    "\n",
    "# place data in a bar chart \n",
    "# where the x-axis is each year and \n",
    "# the y-axis is the number of sales per year\n",
    "ax.bar(x=[sale[0] for sale in sale_year_count],\n",
    "       height=[sale[1] for sale in sale_year_count])\n",
    "\n",
    "# set axis labels\n",
    "ax.set_xlabel(\"Year\")\n",
    "ax.set_ylabel(\"Number of property sales\")\n",
    "\n",
    "# give the bar chart a title\n",
    "ax.set_title(\"Property sales in King County, WA (1900 to 2021)\")\n",
    "\n",
    "# display bar chart clearly\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** What can we learn from this graph?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Level Up: `Counter`\n",
    "\n",
    "The `collections` package has a lot of useful tools for working with Python objects containing multiple elements. You can use the `Counter` class to easily count the number of values in a list.\n",
    "\n",
    "For example, let's say that for all dictionaries in `sales`, we want to count how many property sales occured in each year. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "sale_year_count = Counter([sale['sale_year'] for sale in sales])\n",
    "sale_year_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's sort these by year. Since the value in `sale_year` is an integer, we can use Python's built-in [`sorted`](https://docs.python.org/3/library/functions.html#sorted) function to transform the `Counter` object into a `list` of `tuples` - `(year, count)` which are sorted in ascending order by year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sale_year_count = sorted(sale_year_count.items())\n",
    "sale_year_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Level Up: Using `datetime`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has a [`datetime`](https://docs.python.org/3.7/library/datetime.html) package that is the standard tool for handling dates and times. `datetime` objects make it easy to do fun things like subtract dates to calculate how far apart they are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sale in sales:\n",
    "    # Transform DocumentDate from string to datetime\n",
    "    sale[\"clean_date\"] = datetime.strptime(sale[\"DocumentDate\"], \n",
    "                                           \"%m/%d/%Y\")\n",
    "    \n",
    "    # Add the sale year and month to each dictionary element of sales\n",
    "    sale[\"sale_year\"] = sale[\"clean_date\"].year\n",
    "    sale[\"sale_month\"] = sale[\"clean_date\"].month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect our work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales[0]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}