DataScienceTopics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Topics\n",
    "\n",
    "How to address issues that complicate data modeling and analytics.\n",
    "\n",
    "### Table of Contents\n",
    "1. [Missing Data](#missingdata)\n",
    "1. [Unbalanced Data](#unbalanceddata)\n",
    "1. [Curse of Dimensionality](#curseofdimensionality)\n",
    "1. [Multicollinearity](#multicollinearity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"qs\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "# Questions to ask\n",
    "<br>\n",
    "References:<br>\n",
    "[Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/)\n",
    "___\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When presented with a dataset, what are the relevant questions?\n",
    "\n",
    "Consider a marketing plan where money is spent on 3 independent advertising campaigns on TV, radio, and newspapers, and the resulting sales for the product is tracked for each campaign. So we have sales from TV advertising, sales from advertising, and sales from newspaper advertising.\n",
    "\n",
    "The independent variable is categorical (with 3 values): advertising medium<br>\n",
    "The dependent variable is continuous: sales\n",
    "\n",
    "Important business questions are likely to be:\n",
    "1. **Is there a relationship between advertising sales and budget?**\n",
    " - Use ANOVA and compute f-statistic and p-value then reject/accept null hypothesis of no relationship, meaning multiple regression model has coefficients all equal to zero.<br><br>\n",
    "1. **How strong is the relationship?**\n",
    " - Look at RSE of regression model and $R^2$ statistic.<br><br>\n",
    "1. **Which media campaign contributes to sales?**\n",
    " - Use pairwise t-tests to compare each media coefficient with zero.<br><br>\n",
    "1. **How large is the effect of each medium on sales?**\n",
    " - Report coefficients along with confidence interval. Check for multicollinearity using VIF.<br><br>\n",
    "1. **How accurately can we predict future sales?**\n",
    " - Use confidence or predection intervals for dependent variable.<br><br>\n",
    "1. **Is the relationship linear?**\n",
    " - Examine plots of residuals, if there is no pattern than the effect is linear.<br><br>\n",
    "1. **Is there synergy among the advertising media?**\n",
    " - Compare model performance using $R^2$ with and without interact terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"missingdata\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "# Missing Data\n",
    "<br>\n",
    "References: <br>\n",
    "[https://www.utexas.edu/cola/prc/_files/cs/Missing-Data.pdf](https://www.utexas.edu/cola/prc/_files/cs/Missing-Data.pdf)<br>\n",
    "[https://github.com/hammerlab/fancyimpute](https://github.com/hammerlab/fancyimpute)<br>\n",
    "[Schafer & Graham 2002](http://www.nyu.edu/classes/shrout/G89-2247/Schafer&Graham2002.pdf)<br>\n",
    "[http://cmhsr.wustl.edu/Resources/Documents/Imputing...pdf](http://cmhsr.wustl.edu/Resources/Documents/Imputing%20missing%20data_A%20comparison%20of%20methods%20for%20Social%20Work%20Researchers.pdf)\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of procedure\n",
    "1. Identify relationships in missing data\n",
    " * ***Missing completely at random (MCAR):*** missingness does not depend on missing variable or other variables\n",
    " * ***Missing at random (MAR):*** missingness depends on other variables but not missing variable\n",
    " * ***Missing not at random (MNAR):*** missingness depends on missing variable\n",
    "<br><br>\n",
    "1. Select appropriate method to impute or delete missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Identify relationships\n",
    "\n",
    "First, check relationships between missing data variable and other variables using only examples/rows with non-missing data: \n",
    "- If there is a strong correlation between missing variable and other independent variables then it is unlikely to add any additional predictive power and can be discarded.\n",
    "- If there is a strong correlation between missing variable and dependent variable (output feature) then it should definitely be kept and a suitable method for imputation found.\n",
    "- If there are no strong correlations then the feature should be kept and dealt with appropriately as it may still have some predictive ability. \n",
    "\n",
    "Create dummy boolean/Bernoulli variable to describe missingness:\n",
    "- If missingness correlates strongly with other independent variables then data can be interpreted as MAR.\n",
    "- If missingness correlates strongly with output variable, then this feature may be replaced with the missingness variable. For classification, compare distribution of output values between missing and non-missing examples, if significantly different, then the missingness likely has some predictive power.\n",
    "- To determine MNAR would require some range of potential values to be known.\n",
    "\n",
    "For data MNAR:\n",
    "- Often not possible to know if MNAR is true, unless there is a known range of values that are missing.\n",
    "- Sometimes it is possible to predict missingness using other variables. For example, if high income earners don't report income, they may be identified by education or investments. \n",
    "- It may not be necessary to distinguish MNAR as some imputation methods for MAR also work here. \n",
    "\n",
    "For data MCAR or MAR:\n",
    "- Some of the methods for imputation/deletion will work in either case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Methods for addressing missing values\n",
    "\n",
    "### Not recommended\n",
    "- ***Single imputation with mean or mode:*** reduces variability, weakens correlation and covariance because it ignores relationships between variables including the output variable.\n",
    "\n",
    "### If data are missing completely at random (MCAR)\n",
    "- ***Listwise deletion:*** remove examples (rows) with missing data. Disadvantages include reduced statistical power if data are MCAR, if not MCAR this may introduce bias as the subsample data is not a good representation of the original data.  List-wise deletion can produce unbiased linear regression parameters as long as missingness is not a function of outcome variable. If dataset is small, then statistical power will be lost.\n",
    "\n",
    "### If data are missing at random (MAR)\n",
    "- ***Regression imputation:*** replacing missing variables with regression using other indpendent variables (perhaps those that have a high correlation with a dummy variable expressing missingness). Repeat regression until values between iterations converge. Overestimates model fit and correlation and reduces variability which can hurt predictive power.\n",
    "<br><br>\n",
    "- ***Single imputation with a conditional distribution:*** Use a highly correlated variable $(X)$ to construct a distribution of values for the missing variable $(Y)$ then draw values of $Y$ from this distribution (*conditional probability*),\n",
    "<br><br>\n",
    "$$P(Y_{mis} | Y_{obs};\\theta) = \\frac{P(Y_{obs};\\theta | Y_{mis})}{P(Y_{obs};\\theta)}$$\n",
    "<br>\n",
    "Computing from a conditional distribution is essentially unbiased under MCAR or MAR but potentially biased under MNAR. This method has the drawback of understating uncertainty as the fact that missing values are guesses is not included in the analysis. Although using a conditional distribution for imputation does a better job of preserving uncertainty. <br><br>Single imputation is reasonable if a dataset with many features is missing a small percentage of values that are spread evenly across the dataset.\n",
    "<br><br>\n",
    "- ***Multiple imputation with chained equations (MICE):*** This method involves filling in the missing values multiple times to create several complete datasets. Creating several estimates for missing values better characterizes the error in imputation. <br><br>In MICE, a series of regression models are run whereby each missing value is modeled conditional upon the other variables. So each variable is modeled according to its distribution, binary variables are modeled with logistic regression and continuous with linear regression.<br><br>Here is the process:\n",
    " 1. A simple imputation (mean) is performed for all missing values in the dataset.\n",
    " 1. One of the variables with missing data $(Y)$ is kept missing.\n",
    " 1. The observed values for $Y$ are regressed with the other variables (all or some) in the imputed data.\n",
    " 1. The missing values in $Y$ are then imputed with predictions from this regression.\n",
    " 1. The previous steps are repeated for each variable with missing data.\n",
    " 1. Again, this process is repeated with the just-imputed values until convergence (typically 10 cycles are used).\n",
    " 1. After obtaining one \"complete\" dataset, repeat the entire procedure with missing variables imputed in a different order to obtain additional \"complete\" datasets, typically 5-10.\n",
    " 1. The final analysis is performed on each of these datasets to provide an estimate of the error that arises from the imputation process.\n",
    " <br><br>\n",
    "- ***Soft imputation:*** For datasets with high dimensionality and very few observations. Assumes the parameter describing the missing values $(Z)$ lies in a a much lower dimensional manifold. The problem becomes minimizing the nuclear norm $||Z||_{*}$ subject to the following optimization, problem,\n",
    "$$ \\sum_{(i,j) \\in \\Omega} (X_{ij} - X_{ij})^{2} \\le \\delta $$\n",
    " Which can be expressed in Lagrange form,\n",
    "$$ \\min_{Z} \\sum_{(i,j) \\in \\Omega} (X_{ij} - X_{ij})^{2} + \\lambda \\ ||Z||_{*} $$\n",
    " Soft impute is an algorithm for nuclear norm regularization. It repeatedly replaces missing entries with the current guess, then updates them by solving the Lagrange equation above. This equation is solved with SVD of the truncated dataset that was used to construct $Z$.\n",
    "<br><br>\n",
    "- ***k-nearest neighbors:*** This method is appropriate for discrete and continuous variables. One drawback is the need to search through large datasets (many examples) to find the closest values, this can be mitigated by using a small training set or other techniques. Not well-suited to datasets with high dimensionality.\n",
    "\n",
    "### If data are missing not at random (MNAR)\n",
    "- Try imputation methods appropriate for MAR and see if model behaves as expected.\n",
    "- Otherwise, collect new data.\n",
    "\n",
    "## Appendix\n",
    "\n",
    "### Maximum likelihood for missing data\n",
    "Rather than imputing missing values, maximum likelihood computes parameters that maximize the log likelihood of obtaining the data set. The idea is to select the parameters $\\theta$ that maximize,\n",
    "$$ L = \\prod_{i=1}^{n} f_{i} (y_{i1}, y_{i2},..., y_{ik}; \\theta) $$\n",
    "where $f$ is the joint probability function for observation $i$ and $\\theta$ is the set of parameters to be estimated.\n",
    "\n",
    "For example, if there are $m$ observations with complete data and $m-n$ are missing data for the first two variables (1 and 2), the likelihood to be maximized is,\n",
    "$$ L = \\prod_{i=1}^{m} f_{i} (y_{i1}, y_{i2},..., y_{ik}; \\theta) \\prod_{i=m+1}^{n} f_{i} (y_{i3}, y_{i4},..., y_{ik}; \\theta) $$\n",
    "\n",
    "Once these parameters are determined, a linear model can be constructed such as ordinary least squares.\n",
    "\n",
    "### Summary of single imputation methods\n",
    "<img src=\"images/singleimp.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Python library with additional methods*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# fancyimpute\n",
    "\n",
    "A variety of matrix completion and imputation algorithms implemented in Python.\n",
    "\n",
    "## Usage\n",
    "\n",
    "```python\n",
    "from fancyimpute import BiScaler, KNN, NuclearNormMinimization, SoftImpute\n",
    "\n",
    "# X is the complete data matrix\n",
    "# X_incomplete has the same values as X except a subset have been replace with NaN\n",
    "\n",
    "# Use 3 nearest rows which have a feature to fill in each row's missing features\n",
    "X_filled_knn = KNN(k=3).complete(X_incomplete)\n",
    "\n",
    "# matrix completion using convex optimization to find low-rank solution\n",
    "# that still matches observed values. Slow!\n",
    "X_filled_nnm = NuclearNormMinimization().complete(X_incomplete)\n",
    "\n",
    "# Instead of solving the nuclear norm objective directly, instead\n",
    "# induce sparsity using singular value thresholding\n",
    "X_filled_softimpute = SoftImpute().complete(X_incomplete_normalized)\n",
    "\n",
    "# print mean squared error for the three imputation methods above\n",
    "nnm_mse = ((X_filled_nnm[missing_mask] - X[missing_mask]) ** 2).mean()\n",
    "print(\"Nuclear norm minimization MSE: %f\" % nnm_mse)\n",
    "\n",
    "softImpute_mse = ((X_filled_softimpute[missing_mask] - X[missing_mask]) ** 2).mean()\n",
    "print(\"SoftImpute MSE: %f\" % softImpute_mse)\n",
    "\n",
    "knn_mse = ((X_filled_knn[missing_mask] - X[missing_mask]) ** 2).mean()\n",
    "print(\"knnImpute MSE: %f\" % knn_mse)\n",
    "```\n",
    "\n",
    "## Algorithms\n",
    "\n",
    "* `SimpleFill`: Replaces missing entries with the mean or median of each column.\n",
    "\n",
    "* `KNN`: Nearest neighbor imputations which weights samples using the mean squared difference\n",
    "on features for which two rows both have observed data.\n",
    "\n",
    "* `SoftImpute`: Matrix completion by iterative soft thresholding of SVD decompositions. Inspired by the [softImpute](https://web.stanford.edu/~hastie/swData/softImpute/vignette.html) package for R, which is based on [Spectral Regularization Algorithms for Learning Large Incomplete Matrices](http://web.stanford.edu/~hastie/Papers/mazumder10a.pdf) by Mazumder et. al.\n",
    "\n",
    "* `IterativeSVD`: Matrix completion by iterative low-rank SVD decomposition. Should be similar to SVDimpute from [Missing value estimation methods for DNA microarrays](http://www.ncbi.nlm.nih.gov/pubmed/11395428) by Troyanskaya et. al.\n",
    "\n",
    "* `MICE`: Reimplementation of [Multiple Imputation by Chained Equations](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/).\n",
    "\n",
    "* `MatrixFactorization`: Direct factorization of the incomplete matrix into low-rank `U` and `V`, with an L1 sparsity penalty on the elements of `U` and an L2 penalty on the elements of `V`. Solved by gradient descent.\n",
    "\n",
    "* `NuclearNormMinimization`: Simple implementation of [Exact Matrix Completion via Convex Optimization](http://statweb.stanford.edu/~candes/papers/MatrixCompletion.pdf\n",
    ") by Emmanuel Candes and Benjamin Recht using [cvxpy](http://www.cvxpy.org/en/latest/). Too slow for large matrices.\n",
    "\n",
    "* `BiScaler`: Iterative estimation of row/column means and standard deviations to get doubly normalized\n",
    "matrix. Not guaranteed to converge but works well in practice. Taken from [Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares](http://arxiv.org/abs/1410.2596)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probabilistic methods\n",
    "\n",
    "Impute missing values based on distributions of non-missing values.\n",
    "\n",
    "### Approach\n",
    "\n",
    "1. For feature $x_{missing}$ with missing values, use the distribution of non-missing values for imputation. For example, if feature has non-missing values of 1 five times and 2 three times, randomly select one of these values for imputation.\n",
    "\n",
    "1. Identify patterns between missing feature $x_{missing}$ and non-missing features $x_0,x_1,\\ldots,x_n$ to narrow possible range of values. For example, if row with missing data has non-missing features with values similar to other rows where feature $x_{missing}$ is not missing, than calculate the probability for each possible value and sample from this. Can also use correlations between input features to better determine weights for influence of non-missing feature on missing feature.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "The [numpy random package](https://docs.scipy.org/doc/numpy/reference/routines.random.html) contains a variety of standard distributions which can be sampled and also can sample from custom distributions created from an array of values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"unbalanceddata\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "# Unbalanced Data\n",
    "<br>\n",
    "References:<br>\n",
    "[https://github.com/scikit-learn-contrib/imbalanced-learn](https://github.com/scikit-learn-contrib/imbalanced-learn)<br>\n",
    "[https://www.jair.org/media/953/live-953-2037-jair.pdf](https://www.jair.org/media/953/live-953-2037-jair.pdf)\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most classification algorithms will only perform optimally when the number of samples of each class is roughly the same. Highly skewed datasets, where the minority heavily outnumbered by one or more classes, have proven to be a challenge while at the same time becoming more and more common.\n",
    "\n",
    "One way of addresing this issue is by resampling the dataset as to offset this imbalance with the hope of arriving and a more robust and fair decision boundary than you would otherwise.\n",
    "\n",
    "Resampling techniques are divided in three categories: \n",
    "1. **Under-sampling** the majority class(es)\n",
    "1. **Over-sampling** the minority class\n",
    "1. **Ensemble sampling**.\n",
    "\n",
    "Below is a list of the methods currently implemented in the `imlearn` module.\n",
    "- Under-sampling\n",
    " 1. Random majority under-sampling with replacement\n",
    " 1. Extraction of majority-minority Tomek links\n",
    " 1. Under-sampling with Cluster Centroids\n",
    " 1. NearMiss-(1 & 2 & 3)\n",
    " 1. Condensend Nearest Neighbour\n",
    " 1. One-Sided Selection\n",
    " 1. Neighboorhood Cleaning Rule\n",
    "- Over-sampling\n",
    " 1. Random minority over-sampling with replacement\n",
    " 1. SMOTE - Synthetic Minority Over-sampling Technique\n",
    " 1. bSMOTE(1&2) - Borderline SMOTE of types 1 and 2\n",
    " 1. SVM_SMOTE - Support Vectors SMOTE\n",
    "- Over-sampling followed by under-sampling\n",
    " 1. SMOTE + Tomek links\n",
    " 1. SMOTE + ENN\n",
    "- Ensemble sampling\n",
    " 1. EasyEnsemble\n",
    " 1. BalanceCascade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algorithms\n",
    "\n",
    "### Over-sampling\n",
    "\n",
    "#### Synthetic Minority Oversampling Technique (SMOTE)\n",
    "\n",
    "Creates synthetic samples from the minor class by selecting two or more similar instances (using a distance measure) and perturbing each instance one feature at a time by a random amount that is less than the difference between the neighboring instances.\n",
    "\n",
    "In essence, the synthetic samples are generated along line segments connecting near nearest neighbors. This is done by finding the difference between the vectors representing nearest neighbors then multiplying by a constant between 0 and 1 and adding this amount to the example under consideration.\n",
    "\n",
    "### Under-sampling\n",
    "\n",
    "#### Condensed Nearest Neighbors (CNN)\n",
    "\n",
    "Selects the subset of examples that are able to correctly classify the original dataset using a one-nearest neighbor rule. Given a training set $X$:\n",
    "1. Scan $X$ to find an example $x$ whose nearest example in the undersampled dataset $U$ has a different label from $x$.\n",
    "1. Remove $x$ from $X$ and add to $U$\n",
    "1. Repeat scanning and removing until no more examples are added to $U$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"curseofdimensionality\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "# Curse of dimensionality\n",
    "\n",
    "References:<br>\n",
    "[https://en.wikipedia.org/wiki/Curse_of_dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality)<br>\n",
    "[http://cleverowl.uk/2016/02/06/curse-of-dimensionality-explained/](http://cleverowl.uk/2016/02/06/curse-of-dimensionality-explained/)<br>\n",
    "[http://scikit-learn.org/stable/tutorial/statistical_inference/supervised_learning.html#the-curse-of-dimensionality](http://scikit-learn.org/stable/tutorial/statistical_inference/supervised_learning.html#the-curse-of-dimensionality)\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Increasing the dimensionality of the available data can make it sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. \n",
    "\n",
    "Also organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data however all objects appear to be sparse and dissimilar in many ways which prevents common data organization strategies from being efficient. One example is the nearest neighbors method in machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"multicollinearity\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "___\n",
    "# Multicollinearity\n",
    "\n",
    "References:<br>\n",
    "[Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/)\n",
    "___\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collinearity\n",
    "\n",
    "***Collinearity*** is the situation in which two or more predictor variables are closely related to one another. With high collinearity, a scatter plot of two predictor variables (or features) will resemble a line while low collinearity will show no discernable pattern.\n",
    "\n",
    "For example, when trying to predict the `balance` on a credit card, there may a high degree of collinearity between credit `rating` and credit `limit` as predictors. As a result, there is a larger range of linear regression coefficent values for these two features that yield the minimum sum of squared residuals (RSS), credit `balance`. *In the figure below, notice the difference in scale for `limit` between the left and right, from [0.16,0.19] to [-0.1,0.2].*\n",
    "\n",
    "<table border=\"1\">\n",
    "<tr><td>\n",
    "<img src=\"images/collin.png\" width=\"600\">\n",
    "</td></tr>\n",
    "</table>\n",
    "\n",
    "Collinearity reduces the accuracy of the estimates of the regression coefficients/parameters and causes the standard error for each $\\hat{\\beta}_j$ to grow, this also reduces the t-statistic and power of the hypothesis test, thus increasing the chance for type II error (to incorrectly fail to reject the null hypothesis that $\\beta_j=0$ when the coefficient is actually non-zero).\n",
    "\n",
    "**Detecting collinearity** can be done by examining the correlation matrix of the predictors, an extreme value (near -1 or +1) indicates that two of the variables are correlated and have collinearity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multicollinearity\n",
    "\n",
    "It is possible for three or more variables to have collinearity even if there is not a high correlation between any pair of them, this is ***multicollinearity***.\n",
    "\n",
    "**Detecting multicollinearity** can be done with several methods:\n",
    "1. ***Variance inflation factor*** - after fitting a multiple-variable least square regression to each of the $i$ predictors as a function of the other predictors, calculate VIF for each of these predictors from the coefficient of determination of the regression ($VIF_i = 1 / (1-R_i^2)$). The rule of thumb is if $VIF(\\hat{\\beta}_i) > 10$ then multicollinearity is high. The smallest possible value is 1, indicating absence of multicollinearity. This value can determine the feature that suffers from multicollinearity, then by iteratively removing the other features and calculating the correlation, the offending features can be identified.\n",
    "<br><br>\n",
    "1. ***Condition number test*** - the condition index measures the *ill-conditioning* of a matrix, meaning the inversion of the matrix is numerically unstable and highly sensitive to changes in the original matrix. The condition number is calculated from the maximum eigenvalue divided by the minimum eigenvalue. If the condition number > 30, then multicollinearity exists. This method can also be used to identify which variables are causing the multicollinearity, see this [stack overflow post](http://stackoverflow.com/questions/25676145/capturing-high-multi-collinearity-in-statsmodels).\n",
    "\n",
    "Multicollinearity does not necessarily hurt the accuracy of the prediction, however it will make it difficult to understand how individual predictors are influencing the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Addressing collinearity\n",
    "\n",
    "The two simplest solutions are:\n",
    "1. Drop one of the problematic variables. Since there is duplicate information, this will likely not affect the predictive model. For example, dropping a variable from a linear model may not significantly effect the coefficient of determination $R^2$.\n",
    "1. It may also be possible to combine the collinear variables into one by taking the mean average, for example."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}