Lab 21 - Simple linear regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get an error running the import statements (first coding cell below), run the following cell (it will take 3-4 minutes) to download and install the correct version of statsmodels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user 'statsmodels==0.10.0'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it has finished running (the `[*]` changes to number), restart the kernel by clicking on \"Kernel > Restart\" in the top menu.  Rerun the import statements and there should no longer be an error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 21 - Simple linear regression\n",
    "\n",
    "We will use the dataset with information about the labor market for recent college graduates from Lab 20.  You can also download the CSV file from Blackboard.\n",
    "\n",
    "### 21.1 Loading and cleaning the data\n",
    "\n",
    "We will use a new library in this lab, called statsmodels. Import this and our other usual libraries so we can use them in our code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import statsmodels.formula.api as smf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in Lab 20 we need to read our CSV file into a dataframe and clean it.\n",
    "\n",
    "Load the data in the dataframe `labor`, remembering to skip the non-data rows at the start and end of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your `labor` dataframe below to check it was created properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make things easier, we will rename the columns to shorter names without spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "labor.columns = [\"major\",\"unemployment\",\"underemployment\",\"early\",\"mid\",\"graduate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to remove the commas from the `early` and `mid` columns, and change the column type to float, as in Lab 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "<code>labor[\"early\"] = labor[\"early\"].str.replace(\",\",\"\").astype(float)\n",
    "labor[\"mid\"] = labor[\"mid\"].str.replace(\",\",\"\").astype(float)</code>\n",
    "</details>\n",
    "\n",
    "Check that this code worked correctly by displaying `labor` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 21.2 Simple linear regression\n",
    "\n",
    "Recall from Lab 20 that the columns `early` (which was `Median Wage Early Career`) and `mid` (which was `Median Wage Mid-Career`) were the most correlated. \n",
    "\n",
    "Let's remind ourselves what the relationship looked like by plotting a scatter plot with `early` on the x axis and `mid` on the y axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "<code>df.plot.scatter(x = \"column name 1\", y = \"column name 2\")</code>\n",
    "</details>\n",
    "\n",
    "We can perform linear regression with the following code, which does not display anything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "lm = smf.ols(formula = 'mid ~ early', data = labor).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formula is always `dependent_variable ~ independent_variable`, and in this form, the variable names, which are the column names, cannot have spaces in them or else we need to use `Q('variable name with spaces')`.\n",
    "\n",
    "All the information about the linear model is stored in the variable `lm`.  To see this information, type and run  `lm.summary()` below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a lot of information!  We will only focus on a few of the numbers.  The coefficients for the regression line can be found in the middle section, under the column `coef`: \n",
    "$\\beta_0 = 6394.6948$ and $\\beta_1 = 1.4452$.  \n",
    "\n",
    "Thus, the regression line equation is $y = 1.4452x + 6394.6948$\n",
    "\n",
    "You can also get the coefficients by typing `lm.params`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 21.3  Plotting the regression line and residuals\n",
    "\n",
    "We will use Seaborn to plot the regression line on the scatterplot in one step.  Type and run the code: `sns.regplot(x = 'early',y = 'mid', data = labor)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To further check if the linear model is a good fit for the data, we can plot the residuals to see if they have a normal distribution.  To get the residuals, type `lm.resid` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you figure out how to plot the residuals as a histogram?  They are already stored as a Pandas Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the distribution of the residuals look normal?  Can you find the residual outlier on the scatter plot?  Can you figure out what major that is?  We looked at how to identify outliers in Lab 5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "We find the row of the largest residual<br>\n",
    "<code>max_row = lm.resid.idxmax()</code><br>\n",
    "and then display that row:<br>\n",
    "<code>labor.loc[max_row]</code>\n",
    "</details>\n",
    "\n",
    "### 21.4 Regression line: underemployment vs. median wage early career\n",
    "\n",
    "The next two most correlated columns were `underemployment` and `early`. Compute the linear regression model with `underemployment` as the independent variable and `early` as the dependent variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "<code>lm_name = smf.ols(formula = 'dependent_variable ~ indepedent_variable', data = df_name).fit()</code>\n",
    "</details>\n",
    "\n",
    "Display a summary of your linear regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "<code>lm_name.summary()</code>\n",
    "</details>\n",
    "\n",
    "What is the equation for the linear regression line?\n",
    "\n",
    "Use the Seaborn package to plot the scatter plot of underemployment vs. median wage early career with the regression line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "<code>sns.regplot(x = 'independent_variable' ,y = 'dependent_variable', data = df_name)</code>\n",
    "</details>\n",
    "\n",
    "Plot the residuals as a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Pattern:</summary>\n",
    "<code>pd.Series(lm_name.resid).hist()</code>\n",
    "</details>\n",
    "\n",
    "Is the distribution of the residuals roughly normal?  Is your linear regression model a good representation of the relationship between underemployment and median wage early career?\n",
    "\n",
    "### Challenges\n",
    "- In the green taxi trip data, what is the correlation between trip distance and fare amount?  Compute the linear regression line with trip distance as the independent variable and fare amount as the dependent variable.  Find the equation for the regression line, plot the scatterplot with the line, and check if the residuals are normally distributed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}