[W2_Lab04]FeatEng_PolyReg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Feature Engineering and Polynomial Regression\n",
    "\n",
    "![](./images/C1_W2_Lab07_FeatureEngLecture.PNG)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "In this lab you will:\n",
    "- explore feature engineering and polynomial regression which allows you to use the machinery of linear regression to fit very complicated, even very non-linear functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "You will utilize the function developed in previous labs as well as matplotlib and NumPy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from lab_utils_multi import zscore_normalize_features, run_gradient_descent_feng\n",
    "np.set_printoptions(precision=2)  # reduced display precision on numpy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='FeatureEng'></a>\n",
    "# Feature Engineering and Polynomial Regression Overview\n",
    "\n",
    "Out of the box, linear regression provides a means of building models of the form:\n",
    "$$f_{\\mathbf{w},b} = w_0x_0 + w_1x_1+ ... + w_{n-1}x_{n-1} + b \\tag{1}$$ \n",
    "What if your features/data are non-linear or are combinations of features? For example,  Housing prices do not tend to be linear with living area but penalize very small or very large houses resulting in the curves shown in the graphic above. How can we use the machinery of linear regression to fit this curve? Recall, the 'machinery' we have is the ability to modify the parameters $\\mathbf{w}$, $\\mathbf{b}$ in (1) to 'fit' the equation to the training data. However, no amount of adjusting of $\\mathbf{w}$,$\\mathbf{b}$ in (1) will achieve a fit to a non-linear curve.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='PolynomialFeatures'></a>\n",
    "## Polynomial Features\n",
    "\n",
    "Above we were considering a scenario where the data was non-linear. Let's try using what we know so far to fit a non-linear curve. We'll start with a simple quadratic: $y = 1+x^2$\n",
    "\n",
    "You're familiar with all the routines we're using. They are available in the lab_utils.py file for review. We'll use [`np.c_[..]`](https://numpy.org/doc/stable/reference/generated/numpy.c_.html) which is a NumPy routine to concatenate along the column boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = 1 + x**2\n",
    "X = x.reshape(-1, 1)\n",
    "\n",
    "model_w,model_b = run_gradient_descent_feng(X,y,iterations=1000, alpha = 1e-2)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"no feature engineering\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\");  plt.xlabel(\"X\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, as expected, not a great fit. What is needed is something like $y= w_0x_0^2 + b$, or a **polynomial feature**.\n",
    "To accomplish this, you can modify the *input data* to *engineer* the needed features. If you swap the original data with a version that squares the $x$ value, then you can achieve $y= w_0x_0^2 + b$. Let's try it. Swap `X` for `X**2` below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = 1 + x**2\n",
    "\n",
    "# Engineer features \n",
    "X = x**2      #<-- added engineered feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.reshape(-1, 1)  #X should be a 2-D Matrix\n",
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=10000, alpha = 1e-5)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Added x**2 feature\")\n",
    "plt.plot(x, np.dot(X,model_w) + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! near perfect fit. Notice the values of $\\mathbf{w}$ and b printed right above the graph: `w,b found by gradient descent: w: [1.], b: 0.0490`. Gradient descent modified our initial values of $\\mathbf{w},b $ to be (1.0,0.049) or a model of $y=1*x_0^2+0.049$, very close to our target of $y=1*x_0^2+1$. If you ran it longer, it could be a better match. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Features\n",
    "<a name='GDF'></a>\n",
    "Above, we knew that an $x^2$ term was required. It may not always be obvious which features are required. One could add a variety of potential features to try and find the most useful. For example, what if we had instead tried : $y=w_0x_0 + w_1x_1^2 + w_2x_2^3+b$ ? \n",
    "\n",
    "Run the next cells. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = x**2\n",
    "\n",
    "# engineer features .\n",
    "X = np.c_[x, x**2, x**3]   #<-- added engineered feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=10000, alpha=1e-7)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"x, x**2, x**3 features\")\n",
    "plt.plot(x, X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the value of $\\mathbf{w}$, `[0.08 0.54 0.03]` and b is `0.0106`.This implies the model after fitting/training is:\n",
    "$$ 0.08x + 0.54x^2 + 0.03x^3 + 0.0106 $$\n",
    "Gradient descent has emphasized the data that is the best fit to the $x^2$ data by increasing the $w_1$ term relative to the others.  If you were to run for a very long time, it would continue to reduce the impact of the other terms. \n",
    ">Gradient descent is picking the 'correct' features for us by emphasizing its associated parameter\n",
    "\n",
    "Let's review this idea:\n",
    "- less weight value implies less important/correct feature, and in extreme, when the weight becomes zero or very close to zero, the associated feature is not useful in fitting the model to the data.\n",
    "- above, after fitting, the weight associated with the $x^2$ feature is much larger than the weights for $x$ or $x^3$ as it is the most useful in fitting the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Alternate View\n",
    "Above, polynomial features were chosen based on how well they matched the target data. Another way to think about this is to note that we are still using linear regression once we have created new features. Given that, the best features will be linear relative to the target. This is best understood with an example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = x**2\n",
    "\n",
    "# engineer features .\n",
    "X = np.c_[x, x**2, x**3]   #<-- added engineered feature\n",
    "X_features = ['x','x^2','x^3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax=plt.subplots(1, 3, figsize=(12, 3), sharey=True)\n",
    "for i in range(len(ax)):\n",
    "    ax[i].scatter(X[:,i],y)\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "ax[0].set_ylabel(\"y\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, it is clear that the $x^2$ feature mapped against the target value $y$ is linear. Linear regression can then easily generate a model using that feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling features\n",
    "As described in the last lab, if the data set has features with significantly different scales, one should apply feature scaling to speed gradient descent. In the example above, there is $x$, $x^2$ and $x^3$ which will naturally have very different scales. Let's apply Z-score normalization to our example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0,20,1)\n",
    "X = np.c_[x, x**2, x**3]\n",
    "print(f\"Peak to Peak range by column in Raw        X:{np.ptp(X,axis=0)}\")\n",
    "\n",
    "# add mean_normalization \n",
    "X = zscore_normalize_features(X)     \n",
    "print(f\"Peak to Peak range by column in Normalized X:{np.ptp(X,axis=0)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can try again with a more aggressive value of alpha:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(0,20,1)\n",
    "y = x**2\n",
    "\n",
    "X = np.c_[x, x**2, x**3]\n",
    "X = zscore_normalize_features(X) \n",
    "\n",
    "model_w, model_b = run_gradient_descent_feng(X, y, iterations=100000, alpha=1e-1)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Normalized x x**2, x**3 feature\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature scaling allows this to converge much faster.   \n",
    "Note again the values of $\\mathbf{w}$. The $w_1$ term, which is the $x^2$ term is the most emphasized. Gradient descent has all but eliminated the $x^3$ term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complex Functions\n",
    "With feature engineering, even quite complex functions can be modeled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(0,20,1)\n",
    "y = np.cos(x/2)\n",
    "\n",
    "X = np.c_[x, x**2, x**3,x**4, x**5, x**6, x**7, x**8, x**9, x**10, x**11, x**12, x**13]\n",
    "X = zscore_normalize_features(X) \n",
    "\n",
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=1000000, alpha = 1e-1)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Normalized x x**2, x**3 feature\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Congratulations!\n",
    "In this lab you:\n",
    "- learned how linear regression can model complex, even highly non-linear functions using feature engineering\n",
    "- recognized that it is important to apply feature scaling when doing feature engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}