zehnder:guebelin_HS20)assignment2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 : Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goal\n",
    "\n",
    "Apply the simplest variant of gradient descent (\"vanilla gradient descent\", VGD) to two simple use cases:\n",
    "\n",
    "a. Linear Regression: Test by comparing with the solution of the normal equations.<br>\n",
    "b. Rosenbrock Function: Explore the behavior of the method with different initial values and learning rates.<br>\n",
    "\n",
    "The formulas for the gradient for the two use cases can be found on the slides. These will be helpful to implement vanilla gradient descent. For validating that the formula and the implementation is correct, you also need to implement gradient checking. \n",
    "\n",
    "**Overview**\n",
    "\n",
    "| Exercise | Content | Punkte |\n",
    "| :--- | :--- | :---: |\n",
    "| Exercise 1 | Vanilla Gradient Descent (VGD) | min req for \"pass\", 1 bonus point |\n",
    "| Exercise 2 | Application of VGD to Linear Regression | min req for \"pass\" |\n",
    "| Exercise 3 | Analysis of VGD applied to Linear Regression | min req for \"pass\", 1 bonus point |\n",
    "| Exercise 4 | Application of VGD to Rosenbrock | min req for \"pass\" |\n",
    "| Exercise 5 | Analysis of VGD applied to Rosenbrock | bonus points: 2 |\n",
    "\n",
    "For convenience, I have added some unit test cells (marked as **<span style=\"color:green\">TEST</span>**). If their execution do result in a **Python-Error** (e.g. Assertion Error) then something is probably wrong with your implementation. Don't modify the test cells.\n",
    "\n",
    "Please send your solution by email to [Martin Melchior](mailto:martin.melchior@fhnw.ch).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparations, Imports, Plotting\n",
    "\n",
    "Implement the methods below in <code>Numpy</code>. \n",
    "\n",
    "**Important Remark:** The function variables $x$ are vectors. With fixed numerical values this translates to numpy-arrays. The shape of the numpy arrays Numpy Arrays ist je nach Use Case unterschiedlich. Wir wählen in den Use Cases Lineare Regression und Rosenbrock Shapes $(n,1)$, wobei $n$ die Anzahl Variablen bezeichnet.\n",
    "\n",
    "For plotting we will use <code>Matplotlib</code>. \n",
    "\n",
    "Accordingly, we only need the following two imports: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For checking progress in the optimization process, we want to see how the cost and the learning speed progresses. The following function can be used to achieve that: <code>learningcurve_plots</code> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "def learningcurve_plots(cost_hist, learning_speed_hist, logy=False):\n",
    "    \"\"\"\n",
    "    cost_hist -- history of cost values, as np-array of shape (T,1)\n",
    "    learning_speed_hist -- history of learning speed values, as np-array of shape (T,1)\n",
    "    logy -- if set to True will plot the y axis at logarithmic scale\n",
    "    \"\"\"\n",
    "    plt.figure(1)\n",
    "    T = len(cost_hist)\n",
    "    if logy:\n",
    "        plt.semilogy(np.arange(T),cost_hist,'b-')\n",
    "    else:\n",
    "        plt.plot(np.arange(T),cost_hist,'b-')\n",
    "    plt.title(\"Cost\")\n",
    "        \n",
    "    plt.figure(2)\n",
    "    T = len(learning_speed_hist)\n",
    "    if logy:\n",
    "        plt.semilogy(np.arange(T),learning_speed_hist,'g-')\n",
    "    else:\n",
    "        plt.plot(np.arange(T),learning_speed_hist,'g-')\n",
    "    plt.title(\"Learning Speed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Vanilla Gradient Descent \n",
    "\n",
    "### Implementation\n",
    "\n",
    "#### <span style=\"color:blue\">EXERCISE 1</span> \n",
    "\n",
    "\n",
    "Implement a function for the computation of Vanilla Gradient Descent.\n",
    "The functions should adopt the signature as described below. For passing other parameters to the cost and gradient function use \"keyword arguments\" (<code>kwargs</code>) that can be passed forward to the calculation of the cost and gradient. \n",
    "\n",
    "The implementation can be tested (to some extent) with the subsequent test cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vanilla_gradient_descent( cost, gradient_cost, learningrate, x_initial, max_iter, eps=1.0e-4, **kwargs):\n",
    "    \"\"\"\n",
    "    Performs plain vanilla gradient descent for the cost function with given gradient-function. The variables x of the function are provided as np-arrays.\n",
    "    \n",
    "    Arguments:\n",
    "    cost -- cost function as a function object that accepts as input a np-array (shape of x) and returns a real number.\n",
    "    gradient_cost -- gradient of the cost function that accepts as input a np-array (shape of x) and returns a np-array (shape of x).\n",
    "    learningrate -- learning rate\n",
    "    x_initial -- initial value, np-array with shape of x\n",
    "    max_iter -- maximum number of iterations\n",
    "    eps -- tolerance used for formulating the stopping criterium: here, formulated in terms of the length of the gradient.\n",
    "    kwargs -- further arguments that can passed forward to the call of the cost and gradient function.\n",
    "    \n",
    "    Returns:\n",
    "    xopt -- the optimum value x where the cost function is minimized (within the given tolerance): np-array with shape of x\n",
    "    cost_hist -- history of the values of the cost function seen during the iteration loop: np-array of shape (T,1) where T is the number of iteration needed.\n",
    "    learning_speed_hist -- history of the learning speed where the learning speed given by the norm of the difference between subsequent x-values: np-array of shape (T,1) where T is the number of iteration needed.\n",
    "    \"\"\"\n",
    "    # START YOUR CODE\n",
    "\n",
    "    iterations = 0\n",
    "    x_current = x_initial\n",
    "        \n",
    "    ch_list = np.array([cost(x_current, **kwargs)])\n",
    "    ls = eps + 1\n",
    "    lsh_list = []\n",
    "    while((ls >= eps) and (iterations < max_iter)):\n",
    "        step = learningrate*(gradient_cost(x_current, **kwargs))\n",
    "        x_current = x_current - step\n",
    "               \n",
    "        ch_list = np.append(ch_list, cost(x_current, **kwargs))\n",
    "        ls = np.linalg.norm(step)\n",
    "        \n",
    "        lsh_list= np.append(lsh_list, ls)\n",
    "        iterations += 1\n",
    "\n",
    "    \n",
    "    ch_list = ch_list.reshape(-2,1)\n",
    "    lsh_list = lsh_list.reshape(-2,1)\n",
    "    return x_current, ch_list, lsh_list\n",
    "    \n",
    "    # END YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:green\">TEST</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of iterations: 176, final cost: 0.00000, final speed: 0.00000\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXlklEQVR4nO3df5TcdX3v8efbbEkgqQTMJsSEkEQjt9gjle7h0gPVItzyQ0u43hahVlKlJ8dTvBcv9AhKC/Ueq1KtWk8rbVq4xBZRakVy7tF7QZpKrSVlQSABYgkJIYkhWUCCgiKB9/3j+10yuzubTXZmdma+83ycM+f7nc/3O/N957ub137m8/3O9xuZiSSpWl7V7gIkSc1nuEtSBRnuklRBhrskVZDhLkkVZLhLUgUZ7pJUQYa7elJE/HZEDEbEjyNiZ0R8MyJOaeD9HouI05tZo9QIw109JyIuBT4HfByYBywCvgAsb2NZUlOF31BVL4mIw4EdwHsz8x/qLJ8OXAOcVzbdDFyemS9ExBzgBuAU4GXgQeCtwGrg3cALwEvA/8rMP23xP0XaL3vu6jW/AswAbhln+ZXAScAvAccDJwJ/WC67DNgO9FP0+D8CZGa+B3gc+I3MnGWwqxMY7uo1rwGezMy94yx/N0XPe3dmDgEfBd5TLnsRmA8ck5kvZua/pB991aEMd/Wap4A5EdE3zvLXAltrnm8t2wA+BWwCbouIzRFxRevKlBpjuKvX/BvF2Pi54yz/AXBMzfNFZRuZ+aPMvCwzlwLnAJdGxGnlevbg1VHG671IlZSZeyLiKuAvI2IvcBvFcMvpwKnATcAfRsTdFIF9FfD3ABHxDmAj8Ciwh+Lg6cvlW+8Clk7hP0XaL3vu6jmZ+WfApRQHSoeAbcAHgK8DHwMGgQeA9cC9ZRvAMuBbwI8pPgF8ITPXlss+QfFH4ZmI+IOp+ZdI4/NUSEmqIHvuklRBhrskVZDhLkkVZLhLUgV1xKmQc+bMycWLF7e7DEnqKvfcc8+Tmdlfb1lHhPvixYsZHBxsdxmS1FUiYut4yxyWkaQKMtwlqYIMd0mqIMNdkirIcJekCjLcJamCDHdJqqCuDvcNG+CP/giGhtpdiSR1lq4O940b4WMfg1272l2JJHWWrg73GTOK6U9/2t46JKnTdHW4T59eTF94ob11SFKn6epwt+cuSfV1dbjbc5ek+gx3Saqgrg53h2Ukqb6uDnd77pJUXyXC3Z67JI3U1eE+PCxjz12SRpow3CPi+ojYHREb6iy7LCIyIuaUzyMiPh8RmyLigYg4oRVFD3NYRpLqO5Ce+w3AmaMbI+Jo4NeBx2uazwKWlY+VwLWNlzg+h2Ukqb4Jwz0z7wSerrPos8CHgKxpWw58MQt3AbMjYn5TKq1j2jTo67PnLkmjTWrMPSKWAzsy8/5RixYA22qeby/b6r3HyogYjIjBoQYu6zh9uj13SRrtoMM9Ig4DPgJc1ciGM3NVZg5k5kB/f/+k32fGDHvukjRa3yRe8zpgCXB/RAAsBO6NiBOBHcDRNesuLNtaZvp0w12SRjvonntmrs/MuZm5ODMXUwy9nJCZTwBrgAvLs2ZOAvZk5s7mljzSjBkOy0jSaAdyKuRNwL8Bx0bE9oi4aD+rfwPYDGwC/gb4/aZUuR/23CVprAmHZTLzggmWL66ZT+Dixss6cB5QlaSxuvobquABVUmqp+vD3WEZSRqrEuHusIwkjdT14e6wjCSN1fXhbs9dksbq+nC35y5JY3V9uHtAVZLGqkS4OywjSSN1fbg7LCNJY3V9uNtzl6SxKhHuL71UPCRJha4Pd2+SLUljdX24ex9VSRqr68PdnrskjdX14T7cczfcJWmfyoS7wzKStE/Xh7vDMpI0VteHuz13SRrrQO6hen1E7I6IDTVtn4qIjRHxQETcEhGza5Z9OCI2RcT3I+KMFtX9CsfcJWmsA+m53wCcOartduAXM/NNwH8AHwaIiOOA84E3lq/5QkRMa1q1dTgsI0ljTRjumXkn8PSottsyc2/59C5gYTm/HPhyZr6QmVuATcCJTax3DIdlJGmsZoy5vw/4Zjm/ANhWs2x72TZGRKyMiMGIGBwaGpr0xu25S9JYDYV7RFwJ7AVuPNjXZuaqzBzIzIH+/v5J12DPXZLG6pvsCyPid4F3AKdlZpbNO4Cja1ZbWLa1jAdUJWmsSfXcI+JM4EPAOZn5fM2iNcD5ETE9IpYAy4B/b7zM8TksI0ljTdhzj4ibgF8D5kTEduBqirNjpgO3RwTAXZn5/sx8MCJuBh6iGK65ODNbejFeh2UkaawJwz0zL6jTfN1+1v8T4E8aKepgOCwjSWP5DVVJqqCuD/cIOOQQe+6SVKvrwx28SbYkjVaJcPcm2ZI0UmXC3Z67JO1TiXCfMcOeuyTVqkS423OXpJEqEe4eUJWkkSoR7h5QlaSRKhPu9twlaZ9KhLsHVCVppEqEuz13SRrJcJekCqpEuDssI0kjVSbcf/KTdlchSZ2jEuE+cyY8//zE60lSr6hUuL9yJ1dJ6nEThntEXB8RuyNiQ03bkRFxe0Q8Uk6PKNsjIj4fEZsi4oGIOKGVxQ+bObMIdodmJKlwID33G4AzR7VdAdyRmcuAO8rnAGdR3BR7GbASuLY5Ze7fzJnF9LnnpmJrktT5Jgz3zLwTeHpU83JgdTm/Gji3pv2LWbgLmB0R85tU67gMd0kaabJj7vMyc2c5/wQwr5xfAGyrWW972TZGRKyMiMGIGBwaGppkGQXDXZJGaviAamYmcNCHMjNzVWYOZOZAf39/QzUY7pI00mTDfdfwcEs53V227wCOrllvYdnWUocdVkwNd0kqTDbc1wAryvkVwK017ReWZ82cBOypGb5pGXvukjRS30QrRMRNwK8BcyJiO3A18Eng5oi4CNgKnFeu/g3gbGAT8Dzw3hbUPIbhLkkjTRjumXnBOItOq7NuAhc3WtTBMtwlaaTKfEMVvASBJA2rVLjbc5ekQiXC/dBDIcJwl6RhlQj3iOJ0SMNdkgqVCHcohmYMd0kqGO6SVEGGuyRVkOEuSRVkuEtSBRnuklRBhrskVVBlwt3z3CVpn8qE+8yZXltGkoZVKtztuUtSoVLh/uKLxUOSel2lwh3svUsSGO6SVEkNhXtE/M+IeDAiNkTETRExIyKWRMS6iNgUEV+JiEOaVez+GO6StM+kwz0iFgD/AxjIzF8EpgHnA9cAn83M1wM/BC5qRqETMdwlaZ9Gh2X6gEMjog84DNgJvA34arl8NXBug9s4IIa7JO0z6XDPzB3Ap4HHKUJ9D3AP8Exm7i1X2w4sqPf6iFgZEYMRMTg0NDTZMl5huEvSPo0MyxwBLAeWAK8FZgJnHujrM3NVZg5k5kB/f/9ky3iF4S5J+zQyLHM6sCUzhzLzReBrwMnA7HKYBmAhsKPBGg+I4S5J+zQS7o8DJ0XEYRERwGnAQ8Ba4DfLdVYAtzZW4oEx3CVpn0bG3NdRHDi9F1hfvtcq4HLg0ojYBLwGuK4JdU5oONy9vowkFWe7TFpmXg1cPap5M3BiI+87GYcdVkztuUtShb6h2tcHhxxiuEsSVCjcwStDStKwSoX7rFnw4x+3uwpJar9Khfvhh8Ozz7a7Cklqv0qF++zZ8Mwz7a5CktrPcJekCjLcJamCDHdJqqBKhntmuyuRpPaqXLi//LKnQ0pS5cIdHJqRJMNdkirIcJekCjLcJamCDHdJqiDDXZIqqFLhfvjhxdRwl9TrKhXufX3FZX8Nd0m9rqFwj4jZEfHViNgYEQ9HxK9ExJERcXtEPFJOj2hWsQfCSxBIUuM99z8H/m9m/ifgeOBh4ArgjsxcBtxRPp8yhrskNRDuEXE48BbgOoDM/FlmPgMsB1aXq60Gzm2sxINjuEtSYz33JcAQ8L8j4nsR8bcRMROYl5k7y3WeAObVe3FErIyIwYgYHBoaaqCMkQx3SWos3PuAE4BrM/PNwHOMGoLJzATqXqMxM1dl5kBmDvT39zdQxkizZ8MPf9i0t5OkrtRIuG8HtmfmuvL5VynCfldEzAcop7sbK/Hg2HOXpAbCPTOfALZFxLFl02nAQ8AaYEXZtgK4taEKD9Ls2bBnT3HpX0nqVX0Nvv6/AzdGxCHAZuC9FH8wbo6Ii4CtwHkNbuOg1F7T/dWvnsotS1LnaCjcM/M+YKDOotMaed9G1F6CwHCX1Ksq9Q1V8PoykgSGuyRVkuEuSRVUuXA/orySzdNPt7cOSWqnyoX73LnFdNeu9tYhSe1UuXCfNQtmzjTcJfW2yoU7wLx5hruk3ma4S1IFVTbcn3ii3VVIUvtUNtztuUvqZZUN96eegr17212JJLVHJcP9qKMgE5p4DxBJ6iqVDPd55b2fHJqR1KsMd0mqoEqHu2fMSOpVlQ53e+6SelUlw33WLDj0UMNdUu9qONwjYlpEfC8i/k/5fElErIuITRHxlfIWfFMqojhjxnCX1Kua0XO/BHi45vk1wGcz8/XAD4GLmrCNg+YXmST1sobCPSIWAm8H/rZ8HsDbgK+Wq6wGzm1kG5PlJQgk9bJGe+6fAz4EvFw+fw3wTGYOfzd0O7Cg3gsjYmVEDEbE4FALvm1kz11SL5t0uEfEO4DdmXnPZF6fmasycyAzB/r7+ydbxrjmzYMnn/QSBJJ6U18Drz0ZOCcizgZmAK8G/hyYHRF9Ze99IbCj8TIP3vz5xSUIdu2CBXU/O0hSdU26556ZH87MhZm5GDgf+KfMfDewFvjNcrUVwK0NVzkJxxxTTB97rB1bl6T2asV57pcDl0bEJoox+OtasI0JLV1aTLdsacfWJam9GhmWeUVm/jPwz+X8ZuDEZrxvIxYvLqaGu6ReVMlvqALMmFGMuxvuknpRZcMdYMkSw11SbzLcJamCKh3uS5fCtm3w4ovtrkSSplalw33JEnj5ZXj88XZXIklTq/LhDg7NSOo9hrskVVClw33hQujrM9wl9Z5Kh/u0abBokeEuqfdUOtyhOGNm8+Z2VyFJU6vy4f4LvwAPPlicNSNJvaLy4X788fDccw7NSOotPRHuAPff3946JGkqVT7c3/hGeNWrDHdJvaXy4X7oofCGNxjuknpL5cMdiqEZw11SL+mZcH/sMdizp92VSNLUmHS4R8TREbE2Ih6KiAcj4pKy/ciIuD0iHimnRzSv3MkZPqi6fn1765CkqdJIz30vcFlmHgecBFwcEccBVwB3ZOYy4I7yeVu96U3F1KEZSb1i0uGemTsz895y/kfAw8ACYDmwulxtNXBugzU2bMECmDsX1q1rdyWSNDWaMuYeEYuBNwPrgHmZubNc9AQwrxnbaEQEvPWtsHYtZLa7GklqvYbDPSJmAf8IfDAzn61dlpkJ1I3TiFgZEYMRMTg0NNRoGRM69VTYvh0efbTlm5Kktmso3CPi5yiC/cbM/FrZvCsi5pfL5wO76702M1dl5kBmDvT39zdSxgE59dRiunZtyzclSW3XyNkyAVwHPJyZn6lZtAZYUc6vAG6dfHnNc+yxcNRRhruk3tDXwGtPBt4DrI+I+8q2jwCfBG6OiIuArcB5DVXYJBFF73143D2i3RVJUutMOtwz8zvAeBF52mTft5VOPRVuugk2biwuBSxJVdUT31AddsYZxfSWW9pbhyS1Wk+F+6JFcPLJRe9dkqqsp8Id4IILYMMGL0Ugqdp6Ltx/67eKG2fbe5dUZT0X7nPnwumnw5e+5H1VJVVXz4U7wPveB1u3wte/3u5KJKk1ejLc3/lOeN3r4BOf8FozkqqpJ8O9rw8uvxwGB+Fb32p3NZLUfD0Z7gAXXgivfS1cdZVj75Kqp2fDffp0+PjH4a67YNWqdlcjSc3Vs+EORe/9bW8rhmh+8IN2VyNJzdPT4R4Bf/3X8LOfwbveBS+80O6KJKk5ejrcAV7/erjhBvjOd+D97/fsGUnV0MglfyvjXe+Chx+Gj34UZsyAv/iL4lusktStDPfS1VcXwzKf/CTs2gXXXw+zZ7e7KkmanJ4flhkWUXyp6XOfgzVr4Pjj4bbb2l2VJE2O4T7KJZfAd79bnCp5xhlw1lnwr//qWLyk7mK413HiicUlgT/9aVi3Dk45BX75l+Ezn4EtWwx6SZ2vZeEeEWdGxPcjYlNEXNGq7bTK9Olw2WWwbRtce20xbHPZZbB0KRxzDPzO7xSnUX73u/D00+2uVpJGimxBNzQipgH/AfwXYDtwN3BBZj5Ub/2BgYEcHBxseh3N9sgjxTj8t78Nd95ZHHgd1t8PixfDUUfBvHn7HnPmwMyZ9R+HHlpc52b48So/R0k6CBFxT2YO1FvWqrNlTgQ2ZebmsoAvA8uBuuHeLZYtKx4XX1wMzWzZUpxCuXFj8Xj88eJx992we/fBX7MmYmTYDz+mTSuWDa/TynlJU+v3fg8uvbT579uqcF8AbKt5vh34z7UrRMRKYCXAokWLWlRG60QUQzRLl8Lb3z52+UsvwVNPFY/nnqv/+OlPi/X27p34MfwBq3ba7HlJU2/evNa8b9vOc8/MVcAqKIZl2lVHq0ybVtz1ae7cdlciqRe1apR3B3B0zfOFZZskaQq0KtzvBpZFxJKIOAQ4H1jTom1JkkZpybBMZu6NiA8A/w+YBlyfmQ+2YluSpLFaNuaemd8AvtGq95ckjc8zqyWpggx3Saogw12SKshwl6QKasm1ZQ66iIghYOskXz4HeLKJ5bRaN9Vrra3RTbVCd9Xba7Uek5n99RZ0RLg3IiIGx7twTifqpnqttTW6qVbornqtdR+HZSSpggx3SaqgKoT7qnYXcJC6qV5rbY1uqhW6q15rLXX9mLskaawq9NwlSaMY7pJUQV0d7p18E+6IODoi1kbEQxHxYERcUrb/cUTsiIj7ysfZ7a4VICIei4j1ZU2DZduREXF7RDxSTo9od50AEXFszf67LyKejYgPdsq+jYjrI2J3RGyoaau7L6Pw+fJ3+IGIOKEDav1URGws67klImaX7Ysj4ic1+/evOqDWcX/mEfHhcr9+PyLOmMpa91PvV2pqfSwi7ivbm79vM7MrHxSXEn4UWAocAtwPHNfuumrqmw+cUM7/PMUNw48D/hj4g3bXV6fex4A5o9r+FLiinL8CuKbddY7ze/AEcEyn7FvgLcAJwIaJ9iVwNvBNIICTgHUdUOuvA33l/DU1tS6uXa9D9mvdn3n5f+1+YDqwpMyKae2ud9TyPwOuatW+7eae+ys34c7MnwHDN+HuCJm5MzPvLed/BDxMcW/ZbrIcWF3OrwbObV8p4zoNeDQzJ/sN56bLzDuBp0c1j7cvlwNfzMJdwOyImD8lhVK/1sy8LTP3lk/voriTWtuNs1/Hsxz4cma+kJlbgE0UmTFl9ldvRARwHnBTq7bfzeFe7ybcHRmeEbEYeDOwrmz6QPmR9/pOGeoAErgtIu4pb14OMC8zd5bzTwAtupVvQ85n5H+QTty3MP6+7PTf4/dRfLIYtiQivhcR346IX21XUaPU+5l3+n79VWBXZj5S09bUfdvN4d4VImIW8I/ABzPzWeBa4HXALwE7KT6adYJTMvME4Czg4oh4S+3CLD47dtR5s+UtHM8B/qFs6tR9O0In7st6IuJKYC9wY9m0E1iUmW8GLgW+FBGvbld9pa74mddxASM7JU3ft90c7h1/E+6I+DmKYL8xM78GkJm7MvOlzHwZ+Bum+KPieDJzRzndDdxCUdeu4SGCcrq7fRXWdRZwb2bugs7dt6Xx9mVH/h5HxO8C7wDeXf4xohzieKqcv4diHPsNbSuS/f7MO3K/AkREH/BO4CvDba3Yt90c7h19E+5yTO064OHM/ExNe+146n8FNox+7VSLiJkR8fPD8xQH1DZQ7M8V5WorgFvbU+G4RvR+OnHf1hhvX64BLizPmjkJ2FMzfNMWEXEm8CHgnMx8vqa9PyKmlfNLgWXA5vZU+UpN4/3M1wDnR8T0iFhCUeu/T3V94zgd2JiZ24cbWrJvp/LocQuORp9NcRbKo8CV7a5nVG2nUHz0fgC4r3ycDfwdsL5sXwPM74Bal1KcWXA/8ODwvgReA9wBPAJ8Cziy3bXW1DwTeAo4vKatI/YtxR+cncCLFGO9F423LynOkvnL8nd4PTDQAbVuohivHv69/aty3f9W/n7cB9wL/EYH1Druzxy4styv3wfO6oTfg7L9BuD9o9Zt+r718gOSVEHdPCwjSRqH4S5JFWS4S1IFGe6SVEGGuyRVkOEuSRVkuEtSBf1/GNJ0LGttXxsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbvUlEQVR4nO3de5gddZ3n8fenu9OdS3c6l86NJiGBBJBLlEwkyKqTlSAEucsqjC7iyDI6so7LMLPsMF7GVVdm19lnEIUF4eEiK4iCG5/BBRUcdBVMB0NIgEiDgaRzT8itc+30d/841eR0pzvdSU53ddX5vJ6nnq5T9TtV39TpfM6vf6dOlSICMzPLvoq0CzAzs9JwoJuZ5YQD3cwsJxzoZmY54UA3M8sJB7qZWU440C33JL1P0vK06+hPku6V9NW067B0OdCtX0laIWlemjVExK8i4qT+2LakUyU9KWmzpC2SFkm6oD/2ZdYbB7plnqTKFHf/E+BnwERgPPA5YFuK9VgZc6BbKiRVSLpJ0muSNkn6gaQxResfkbRW0lZJz0g6tWjdvZJul/S4pFbg3yZ/CdwoaUnynIclDU3az5W0quj5PbZN1v+tpDWSVku6VlJImt7Nv6EBmAbcFRF7k+n/RcSvi/cr6e8kbUz2+7Gi59dI+h+S3pS0TtIdkoYVrb9Q0uKk5/8bSTOL1p0h6XlJ2yU9DAzFyp4D3dLyH4FLgT8FjgHeAr5dtP6nwAwKvd7ngQe7PP/PgK8BdcCvk2UfAc6nELIzgWsOsf9u20o6H7gBmAdMB+YeYhubgGbge5IulTShmzYTgQagEfgEcKekjuGfbwAnAu9K9tUIfDGp4wzgHuAvgLHA/wIWJG8C1cCPgQeAMcAjwIcPUaeVi4jw5KnfJmAFMK+b5S8D5xQ9ngTsA6q6aTsKCKA+eXwvcH83+/l40eN/BO5I5ucCq/rY9h7gvxWtm57se3oP/75jgduA14B24BlgRtF+24ARRe1/AHwBENAKnFC07j3AH5P524H/2mVfyym8Ab4fWA2oaN1vgK+m/Xp7SndyD93SchzwWDKcsIVCwO8HJkiqlPSNZDhmG4UAhkJPt8PKbra5tmh+J1B7iP331PaYLtvubj9vi4hVEXF9RJyQ/JtagfuLmrwVEa1Fj99I9jEOGA4sKjoG/zdZTrKtv+5Yl6yfnDz3GKAlIqLLdq3MOdAtLSuB+RExqmgaGhEtFIZTLqEw7FEPTE2eo6Ln99dlQtdQ6HV3mNzXJ0bESgrDRqcVLR4taUTR4ykUetcbgV3AqUX//vqI6HhjWQl8rcvxGR4R309qbJSkLtu1MudAt4EwRNLQoqkKuAP4mqTjACSNk3RJ0r4O2ENhjHo48PUBrPUHwCclvUPScArDI92SNFrSP0iannzI2wD8OfBsl6b/IKla0vuAC4FHIqIduAv4n5LGJ9trlHRe8py7gE9LmqOCEZI+JKkO+C2FoZzPSRoi6XLgzBIeA8soB7oNhMcp9EY7pi8D/wwsAJ6UtJ1CCM5J2t9PYQihBXiJgwOy30TET4FbgacpfODZse893TTfS+Gvh59TOFVxadLumqI2ayl84Luawge7n46IV5J1/7ljH8nQ0s+Bk5I6moD/QGF8/q2k3TXJur3A5cnjzcBHgUeP/F9teaHOw3BmVkzSOygEdU1EtB3mc+cC34uIY3tpalYS7qGbdSHpsuT0wNHALcBPDjfMzdLgQDc72F8A6ymcirgf+Ey65Zj1jYdczMxywj10M7OcqEprxw0NDTF16tS0dm9mlkmLFi3aGBHjuluXWqBPnTqVpqamtHZvZpZJknr8VrCHXMzMcsKBbmaWEw50M7OccKCbmeWEA93MLCcc6GZmOeFANzPLicwF+tL1S/n7p/6ejTs3pl2KmdmgkrlAX75xOV/71ddYvX112qWYmQ0qmQv0upo6ALbv2Z5yJWZmg0v2Ar26EOg79u5IuRIzs8Elc4FeW124h+72ve6hm5kVy1ygdwy5uIduZtZZ5gL97R66x9DNzDrJXKB7DN3MrHuZC/SaqhqGVAzxGLqZWReZC3QoDLt4yMXMrLNMBnpdTR079nnIxcysWCYD3T10M7ODZTLQ66rr/KGomVkXmQz02upafyhqZtZFJgO9rqbOQy5mZl1kM9A95GJmdpBMBrqHXMzMDpbJQHcP3czsYJkM9NrqWna37aatvS3tUszMBo1eA13SZElPS3pJ0jJJf9VNG0m6VVKzpCWSZvVPuQW+yYWZ2cH60kNvA/46Ik4BzgI+K+mULm3mAzOS6Trg9pJW2YUv0GVmdrBeAz0i1kTE88n8duBloLFLs0uA+6PgWWCUpEklrzbhm1yYmR3ssMbQJU0FzgCe67KqEVhZ9HgVB4c+kq6T1CSpacOGDYdZ6gG+yYWZ2cH6HOiSaoEfAZ+PiG1HsrOIuDMiZkfE7HHjxh3JJgDf5MLMrDt9CnRJQyiE+YMR8Wg3TVqAyUWPj02W9YuOMXQPuZiZHdCXs1wE3A28HBH/1EOzBcDVydkuZwFbI2JNCevsxEMuZmYHq+pDm38D/HvgRUmLk2V/B0wBiIg7gMeBC4BmYCfwyZJXWsRDLmZmB+s10CPi14B6aRPAZ0tVVG982qKZ2cEy+U3R4UOGI+QxdDOzIpkMdEm+a5GZWReZDHRI7ivqIRczs7dlNtB9CV0zs84yG+i+hK6ZWWeZDXT30M3MOstsoHsM3cyss8wGus9yMTPrLLOBXldd5yEXM7MimQ30+pp6tu7emnYZZmaDRmYDfdTQUexq28Xe/XvTLsXMbFDIdKAD7qWbmSUyG+j1Q+sB2LJ7S7qFmJkNEpkN9I4eugPdzKwg84G+dY+HXMzMIMOBXl/jIRczs2KZDXQPuZiZdZb5QPdZLmZmBZkN9NrqWipU4R66mVkis4Euifqaege6mVkis4EOhWEXn+ViZlaQ6UCvH+oeuplZh0wH+qihoxzoZmYJB7qZWU5kOtDra+o9hm5mlsh0oLuHbmZ2QOYDfduebexv3592KWZmqct0oHdcz8W3ojMzy3ig+3ouZmYHONDNzHIiF4HuC3SZmWU80H0bOjOzAzId6B5yMTM7IBeB7i8XmZllPNBH1owE3EM3M4M+BLqkeyStl7S0h/VzJW2VtDiZvlj6MrtXVVFFbXWtA93MDKjqQ5t7gduA+w/R5lcRcWFJKjpM/vq/mVlBrz30iHgG2DwAtRyR0UNH89but9Iuw8wsdaUaQ3+PpBck/VTSqT01knSdpCZJTRs2bCjJjscOH8vGnRtLsi0zsywrRaA/DxwXEe8EvgX8uKeGEXFnRMyOiNnjxo0rwa6hYXgDm3ZuKsm2zMyy7KgDPSK2RcSOZP5xYIikhqOurI/GDnMP3cwMShDokiZKUjJ/ZrLNAesyNwxvYNOuTbRH+0Dt0sxsUOr1LBdJ3wfmAg2SVgFfAoYARMQdwBXAZyS1AbuAKyMi+q3iLsYOG0t7tLN191ZGDxs9ULs1Mxt0eg30iLiql/W3UTitMRUNwwujOxt3bnSgm1lZy/Q3RaFwlgvApl3+YNTMylvmA724h25mVs5yE+g+ddHMyl3mA33ssMKQi3voZlbuMh/oI2tGUlVR5TF0Myt7mQ90Sf5ykZkZOQh0OPDlIjOzcpaLQPcFuszMchLovkCXmVlOAt1j6GZmOQn0jjH0AbyEjJnZoJOLQB87bCxt7W1s27Mt7VLMzFKTi0D31//NzHIS6L5Al5lZTgLdPXQzs5wEuq/nYmaWk0CfUDsBgPWt61OuxMwsPbkI9LrqOoZVDWPN9jVpl2JmlppcBLokJtVNYm3r2rRLMTNLTS4CHWBi7UTW7nCgm1n5cqCbmeVEfgJ9xESPoZtZWctNoE+qm8Rbu99iT9uetEsxM0tFbgJ9Yu1EANa1rku5EjOzdOQu0D3sYmblKjeBPql2EoA/GDWzspWbQO/ooTvQzaxc5SbQx48YDzjQzax85SbQh1QOoWF4A2t2eAzdzMpTbgIdCuPo7qGbWbnKVaD726JmVs5yF+gecjGzcpWrQO8YcomItEsxMxtwuQr0ibUT2bt/L1t2b0m7FDOzAZerQD+m7hgAVm1blXIlZmYDr9dAl3SPpPWSlvawXpJuldQsaYmkWaUvs2+m1E8BYOW2lWmVYGaWmr700O8Fzj/E+vnAjGS6Drj96Ms6Mh2B/ubWN9MqwcwsNb0GekQ8A2w+RJNLgPuj4FlglKRJpSrwcEysnUhVRZUD3czKUinG0BuB4jGOVcmyAVdZUcmxI491oJtZWRrQD0UlXSepSVLThg0b+mUfx9Uf50A3s7JUikBvASYXPT42WXaQiLgzImZHxOxx48aVYNcHm1I/hTe2vtEv2zYzG8xKEegLgKuTs13OArZGRGpf15xSP4WWbS20tbelVYKZWSqqemsg6fvAXKBB0irgS8AQgIi4A3gcuABoBnYCn+yvYvtiSv0U9sd+1mxfw+T6yb0/wcwsJ3oN9Ii4qpf1AXy2ZBUdpeJTFx3oZlZOcvVNUfC56GZWvnIX6JNHFnrlDnQzKze5C/S6mjpGDx3tQDezspO7QIfCsMub2xzoZlZechvob2zxuehmVl5yGejTRk3j9bde940uzKys5DLQTxx7Iq37Wn07OjMrK7kM9BljZwDw6qZXU67EzGzg5DPQxySBvtmBbmblI5eBPqV+CtWV1e6hm1lZyWWgV1ZUcvzo491DN7OykstAh8KwiwPdzMpJrgO9eXMz7dGedilmZgMit4F+4tgT2d22m5Zt3d5rw8wsd3Ib6B2nLv5h0x9SrsTMbGDkN9B96qKZlZncBnrjyEaGVg31qYtmVjZyG+gVquDkhpNZtmFZ2qWYmQ2I3AY6wMwJM3lx/Ytpl2FmNiByHeinjz+d1dtXs2nnprRLMTPrd7kO9JkTZgK4l25mZaEsAn3JuiUpV2Jm1v9yHegTRkygYXiDA93MykKuA12SPxg1s7KR60CHwgejS9cv9TVdzCz3ch/oMyfMZOe+nbz+1utpl2Jm1q/KItABFq9dnG4hZmb9LPeBfvr406murGZhy8K0SzEz61e5D/SaqhrOmHgGz7Y8m3YpZmb9KveBDnDWsWexsGUhbe1taZdiZtZvyiLQ5zTOYVfbLl5c59MXzSy/yiLQzzr2LACeXeVhFzPLr7II9KmjpjJ+xHiea3ku7VLMzPpNWQS6JOY0znEP3cxyrSwCHQrDLss3LWfjzo1pl2Jm1i/KJtDnTp0LwC9X/DLVOszM+kufAl3S+ZKWS2qWdFM366+RtEHS4mS6tvSlHp13H/Nuaqtr+fnrP0+7FDOzflHVWwNJlcC3gXOBVcBCSQsi4qUuTR+OiOv7ocaSGFI5hLlT5/KLP/4i7VLMzPpFX3roZwLNEfF6ROwFHgIu6d+y+se8afNo3tzMii0r0i7FzKzk+hLojcDKoserkmVdfVjSEkk/lDS5uw1Juk5Sk6SmDRs2HEG5R2fe8fMA+MXr7qWbWf6U6kPRnwBTI2Im8DPgvu4aRcSdETE7ImaPGzeuRLvuu1PGncLE2okedjGzXOpLoLcAxT3uY5Nlb4uITRGxJ3n4XeBPSlNeaUni3OPP5YnXnvB1Xcwsd/oS6AuBGZKmSaoGrgQWFDeQNKno4cXAy6UrsbQuPflSNu/azDNvPJN2KWZmJdVroEdEG3A98ASFoP5BRCyT9BVJFyfNPidpmaQXgM8B1/RXwUfrvBPOY2jVUB57+bG0SzEzKylFRCo7nj17djQ1NaWy78sevoyFLQt58z+9SYXK5rtVZpYDkhZFxOzu1pVlml128mW0bG+haXU6byhmZv2hLAP9whMvpKqiih++9MO0SzEzK5myDPQxw8Ywf/p8vrfkez7bxcxyoywDHeCT7/oka3as4cnXnky7FDOzkijbQP/QiR+iYXgD9/z+nrRLMTMribIN9OrKaj5++sdZsHyBr5FuZrlQtoEO8KlZn2Jf+z7ufv7utEsxMztqZR3op40/jXOmncOtv7uVvfv3pl2OmdlRKetAB7jx7BtZvX01Dy19KO1SzMyOStkH+nknnMep407lm7/9Jml9a9bMrBTKPtAl8Tdn/w1L1i3hsVd8fRczy66yD3SAj838GO9oeAc3P3Wzv2hkZpnlQAeqKqr46ge+yisbX+GBFx5IuxwzsyPiQE9cdvJlnNl4Jjc/dTPb9mxLuxwzs8PmQE9I4lvzv8XaHWv5wlNfSLscM7PD5kAvcmbjmXxm9me4beFtLFq9KO1yzMwOiwO9i6+f83UmjJjA1T++ml37dqVdjplZnznQu6gfWs99l97HSxte4sYnb0y7HDOzPnOgd+PcE87lhrNu4DtN3+GRZY+kXY6ZWZ840Hvw9XO+ztmTz+YTP/4Ev1/z+7TLMTPrlQO9BzVVNTz6kUdpGN7AxQ9dzJtb30y7JDOzQ3KgH8KE2gn85KqfsH3PdubdP491O9alXZKZWY8c6L1458R38vjHHqdlewsfuP8DrN6+Ou2SzMy65UDvg7Mnn82//Nm/8ObWN3nvPe+leXNz2iWZmR3Egd5Hc6fO5amrn2Lbnm3M+e4cnv7j02mXZGbWiQP9MLy78d08d+1zTKydyLkPnMstv76F9mhPuywzM8CBfthOGHMCv/3Ub7n8HZdz0y9u4twHzvUQjJkNCg70IzCyZiQPX/Ewd110FwtbFnLad07jK//6Ffa07Um7NDMrYw70IySJa2ddyyvXv8KlJ1/Kl375JU6//XR+9NKPPAxjZqlwoB+lY+qO4aErHuKJjz9BhSq44pErmHn7TB5a+hD72/enXZ6ZlREHeol88IQPsuwvl/Hg5Q/SHu1c9aOrOOm2k7jl17ewdsfatMszszKgtO50P3v27Ghqakpl3/2tPdp59OVH+dbvvsUzbzxDVUUVF514EVeediXzp8+nrqYu7RLNLKMkLYqI2d2uc6D3r+Ubl3P37+/mvhfuY33reqorq5l3/DwuOvEiPjDtA8wYMwNJaZdpZhnhQB8E9rfv5zcrf8NjrzzGY688xootKwBorGtk7tS5zGmcw6xJs3jnxHdSW12bbrFmNmg50AeZiODVza/y9B+f5ukVT/PLFb9kXWvhwl9CnNRwErMmzeKUhlOYPmY6M8bOYPqY6YysGZly5WaWtqMOdEnnA/8MVALfjYhvdFlfA9wP/AmwCfhoRKw41DbLOdC7igjW7FjD82ue7zSt3LayU7vxI8YzddRUGusaOabuGBrrGmkc2cjE2omMHTaWMcPGMHb4WOqq6zyMY5ZTRxXokiqBPwDnAquAhcBVEfFSUZu/BGZGxKclXQlcFhEfPdR2Hei9a93bymtvvUbz5maaNzfz6qZXeWPrG6zevpqW7S1s2b2l2+dVqvLtcB81dBS11bWMGDKi88/qA4+HDRlGdWX121NNZU2nx28vryosr6qoolKVVKiCyorKTvMVqqBSlVRWVCLkNxazEjtUoFf14flnAs0R8XqysYeAS4CXitpcAnw5mf8hcJskRVrjOTkxonoEMyfMZOaEmd2u37lvJ6u3r2btjrVs3rWZzbs2s2nnpgPzuzaxZfcWWve1sr51Pa17W9mxdwet+1pp3dtK0P8vT4UqOoV8x3yFKt4O++LgF8lPqdP80bTr63PyJI9vpHl6ja6ddS03vOeGkm+3L4HeCBT/7b8KmNNTm4hok7QVGAtsLG4k6TrgOoApU6YcYcnWYfiQ4UwfM53pY6Yf9nMjgl1tu9ixdwe723azd//eTtOetj2dH+8/8Hjf/n3sj/20Rzv725OfRY+7W9ddO4Ag6Hjf73iDiYhO831qd5TbyZOBeKMeaHl7jSaMmNAv2+1LoJdMRNwJ3AmFIZeB3Ld1JonhQ4YzfMjwtEsxsxLpyzdFW4DJRY+PTZZ120ZSFVBP4cNRMzMbIH0J9IXADEnTJFUDVwILurRZAHwimb8CeMrj52ZmA6vXIZdkTPx64AkKpy3eExHLJH0FaIqIBcDdwAOSmoHNFELfzMwGUJ/G0CPiceDxLsu+WDS/G/h3pS3NzMwOh6+2aGaWEw50M7OccKCbmeWEA93MLCdSu9qipA3AG0f49Aa6fAt1kMtSvVmqFbJVb5ZqhWzVm6Va4ejqPS4ixnW3IrVAPxqSmnq6OM1glKV6s1QrZKveLNUK2ao3S7VC/9XrIRczs5xwoJuZ5URWA/3OtAs4TFmqN0u1QrbqzVKtkK16s1Qr9FO9mRxDNzOzg2W1h25mZl040M3MciJzgS7pfEnLJTVLuinteopJmizpaUkvSVom6a+S5V+W1CJpcTJdkHatHSStkPRiUldTsmyMpJ9JejX5OXoQ1HlS0fFbLGmbpM8PpmMr6R5J6yUtLVrW7bFUwa3J7/ESSbMGQa3/XdIrST2PSRqVLJ8qaVfRMb5jIGs9RL09vvaS/ktybJdLOm8Q1PpwUZ0rJC1Olpf22EZEZiYKl+99DTgeqAZeAE5Ju66i+iYBs5L5Ogo31z6Fwv1Wb0y7vh5qXgE0dFn2j8BNyfxNwC1p19nN78Fa4LjBdGyB9wOzgKW9HUvgAuCngICzgOcGQa0fBKqS+VuKap1a3G4QHdtuX/vk/9wLQA0wLcmMyjRr7bL+m8AX++PYZq2H/vYNqyNiL9Bxw+pBISLWRMTzyfx24GUK91vNmkuA+5L5+4BL0yulW+cAr0XEkX7TuF9ExDMU7gdQrKdjeQlwfxQ8C4ySNGlACqX7WiPiyYhoSx4+S+HuZINCD8e2J5cAD0XEnoj4I9BMITsGxKFqVeHu3R8Bvt8f+85aoHd3w+pBGZiSpgJnAM8li65P/pS9ZzAMYRQJ4ElJi5KbeANMiIg1yfxaoH/uaHvkrqTzf4jBemyh52M52H+X/5zCXxAdpkn6vaR/lfS+tIrqRnev/WA+tu8D1kXEq0XLSnZssxbomSCpFvgR8PmI2AbcDpwAvAtYQ+FPrsHivRExC5gPfFbS+4tXRuHvwkFzbmtyG8SLgUeSRYP52HYy2I5lTyTdDLQBDyaL1gBTIuIM4Abgf0samVZ9RTLz2he5is6dkZIe26wFel9uWJ0qSUMohPmDEfEoQESsi4j9EdEO3MUA/vnXm4hoSX6uBx6jUNu6jj//k5/r06vwIPOB5yNiHQzuY5vo6VgOyt9lSdcAFwIfS96ASIYuNiXziyiMSZ+YWpGJQ7z2g/XYVgGXAw93LCv1sc1aoPflhtWpScbH7gZejoh/KlpePDZ6GbC063PTIGmEpLqOeQofii2l802/PwH8n3Qq7FanHs5gPbZFejqWC4Crk7NdzgK2Fg3NpELS+cDfAhdHxM6i5eMkVSbzxwMzgNfTqfKAQ7z2C4ArJdVImkah3t8NdH3dmAe8EhGrOhaU/NgO1Ce/JfwE+QIKZ4+8Btycdj1dansvhT+plwCLk+kC4AHgxWT5AmBS2rUm9R5P4WyAF4BlHccTGAv8AngV+DkwJu1ak7pGAJuA+qJlg+bYUnijWQPsozBu+6mejiWFs1u+nfwevwjMHgS1NlMYe+743b0jafvh5PdjMfA8cNEgObY9vvbAzcmxXQ7MT7vWZPm9wKe7tC3psfVX/83MciJrQy5mZtYDB7qZWU440M3McsKBbmaWEw50M7OccKCbmeWEA93MLCf+P1YfDpmf5iwEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#########\n",
    "# TEST\n",
    "#######\n",
    "\n",
    "def test_cost(x, a, b):\n",
    "    \"\"\"\n",
    "    x - the variables to be optimized\n",
    "    a - parameter (treated by vanilla_gradient_descent as kwargs)\n",
    "    b - parameter (treated by vanilla_gradient_descent as kwargs)\n",
    "    \"\"\"\n",
    "    return (a*x[0,0]**2 + b*x[1,0]**2)/2.0\n",
    "\n",
    "def test_gradient_cost(x, a, b):\n",
    "    \"\"\"\n",
    "    x - the variables to be optimized\n",
    "    a - parameter (treated by vanilla_gradient_descent as kwargs)\n",
    "    b - parameter (treated by vanilla_gradient_descent as kwargs)\n",
    "    \"\"\"\n",
    "    return np.array([a,b]).reshape(2,1)*x\n",
    "\n",
    "learningrate = 0.1\n",
    "x0 = np.array([10,10]).reshape(2,1)\n",
    "\n",
    "xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(test_cost, test_gradient_cost, learningrate, x0, 1000, eps=1.0e-8, a=1, b=2)\n",
    "T = cost_hist.size-1\n",
    "print(\"Number of iterations: %i, final cost: %6.5f, final speed: %6.5f\"%(T, cost_hist[-1,0], learning_speed_hist[-1,0]))\n",
    "learningcurve_plots(cost_hist, learning_speed_hist)\n",
    "\n",
    "np.testing.assert_array_equal((2,1), xopt.shape)\n",
    "np.testing.assert_array_equal((T+1,1), cost_hist.shape)\n",
    "np.testing.assert_array_equal((T,1), learning_speed_hist.shape)\n",
    "np.testing.assert_array_almost_equal(np.linalg.norm(xopt), 0.0, decimal=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Application to Linear Regression\n",
    "\n",
    "Apply Vanilla Gradient Descent to the cost function used for linear regression.<br>\n",
    "Test the method by comparing with the solution of the normal equations on the basis of the dataset studied in the lecture and exercise 2 of worksheet week 6. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAO10lEQVR4nO3dX4yldX3H8fdHRgK7VTGADQXt6g3Rmog6QRRFImoUiW28qSY2qWmzNRoDmsbUJs0cLrxoYhp71aRZqjQqBEFujCGYCooXrjkLa1mEGP8Agn/2mPKnqBHQby/OGdiSmTnP2X3OPL+deb+SyZ6z5zm//eyZmc88851z5peqQpLUrucNHUCStDWLWpIaZ1FLUuMsaklqnEUtSY1bWcaiZ511Vu3bt28ZS0vSjnTo0KFfVdXZG922lKLet28f4/F4GUtL0o6U5IHNbnP0IUmNs6glqXEWtSQ1zqKWpMZZ1JLUuE5FneTKJEeS3JPkqiVnknad0WjoBOrDst6Pmffb85K8GrgeuBB4ErgF+HBV/XCz+6yurpZPz5O6S8BfZHnyO5H3Y5JDVbW60W1dzqhfCRysqt9U1dPAN4H3HV8USdKiuhT1EeAtSc5Msge4HHjpcw9Ksj/JOMl4Mpn0nVPacUaj6RlYMr2+ftkxyMllO96Pc0cf0384fwN8BPg1cA/wu6q6arPjHX1Ii3H0sTMMOfqgqq6pqtdX1SXAI8APji+KJGlRnX7XR5KXVNXRJC9jOp++aLmxpN1lbW3oBOrDst6PXUcfdwBnAk8Bn6iq/9rqeEcfkrSYrUYfnc6oq+ot/UaSJHXlKxMlqXEWtSQ1zqKWpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1JjbOoJalxFrUkNc6ilqTGWdSS1DiLWpIaZ1FLUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxnUq6iQfT3JPkiNJrkty2rKDSRreaDR0go3ttlxzdyFPci7wbeBVVfXbJDcAX6uqz292H3chl3aGBOZUxCB2Yq6tdiHvOvpYAU5PsgLsAX52fFEkSYuaW9RV9TDwGeBB4OfAY1V163OPS7I/yTjJeDKZ9J9U0rYYjaZnhsn0+vrloccNuzlXl9HHi4GbgL8EHgW+DNxYVV/Y7D6OPqSdYSeOGJZpyNHH24GfVNWkqp4CvgK86fiiSJIW1aWoHwQuSrInSYDLgHuXG0tSC9bWhk6wsd2Wa+7oAyDJ1UxHH08DdwF/W1W/2+x4Rx+StJitRh8rXRaoqjWg0a9hkrSz+cpESWqcRS1JjbOoJalxFrUkNc6ilqTGWdSS1DiLWpIaZ1FLUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxlnUktQ4i1qSGmdRS1LjLGpJapxFLUmNs6glqXEWtSQ1bm5RJzk/yeFj3h5PctU2ZJN2jdFo6AQnl932eKWquh+cnAI8DLyhqh7Y7LjV1dUaj8c9xJN2hwQW+FTc9Xbi45XkUFWtbnTboqOPy4AfbVXSkqR+LVrU7weu2+iGJPuTjJOMJ5PJiSeTdrjRaHpmmEyvr1/ebd/Wd7WbH6/Oo48kpwI/A/6sqn651bGOPqTF7MRv5ZdpJz5efY0+3g3cOa+kJUn9WqSoP8AmYw9JJ2ZtbegEJ5fd9nh1Gn0k2Qs8CLyiqh6bd7yjD0lazFajj5UuC1TVr4Eze00lSerEVyZKUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxlnUktQ4i1qSGmdRS1LjLGpJapxFLUmNs6glqXEWtSQ1zqKWpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1JjetU1EnOSHJjkvuS3JvkjcsOJq0bjdpcS9ouXXchvxa4o6oOJDkV2FNVj252vLuQq08JdPgw3fa1pD6d0C7kSV4EXAL8NUBVPQk82WdASdLmuow+Xg5MgM8luSvJgSR7n3tQkv1JxknGk8mk96DaXUaj6dlvMr2+fvl4Rhd9riUNYe7oI8kq8B3g4qo6mORfgcer6p82u4+jD/XJ0Yd2g61GH13OqB8CHqqqg7PrNwKv6yucJGlrc4u6qn4B/DTJ+bO/ugz4/lJTScdYW2tzLWm7dH3WxwXAAeBU4MfAh6rqkc2Od/QhSYs5oWd9AFTVYWDDBSRJy+UrEyWpcRa1JDXOopakxlnUktQ4i1qSGmdRS1LjLGpJapxFLUmNs6glqXEWtSQ1zqKWpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1JjbOoJalxFrUkNc6ilqTGdSrqJPcnuTvJ4STuWqu5RqOhE2ys1VzSVrruQn4/sFpVv+qyqLuQK4EOH1rbrtVc0la7kDv6kKTGdS3qAm5NcijJ/o0OSLI/yTjJeDKZ9JdQJ43RaHrGmkyvr18eetzQai6pq66jj3Or6uEkLwG+Dnysqr612fGOPtTqiKHVXNIJjz6q6uHZn0eBm4EL+4snSdrK3KJOsjfJC9YvA+8Ejiw7mE5ua2tDJ9hYq7mkrcwdfSR5BdOzaIAV4EtV9emt7uPoQ5IWs9XoY2Xenavqx8Brek8lSerEp+dJUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxlnUktQ4i1qSGmdRS1LjLGpJapxFLUmNs6glqXEWtSQ1zqKWpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1Jjetc1ElOSXJXkq8uM9BuMBr1t9all/a3Vp/6/D9Ku93cXcifOTD5BLAKvLCqrtjqWHch31oCHR/2bV2rT63mklq11S7knc6ok5wHvAc40GcwSdJ8XUcfnwU+CfxhswOS7E8yTjKeTCZ9ZNtRRqPpWWYyvb5++XhGBJdeuvFaQ49B+vw/SnrW3NFHkiuAy6vqI0kuBf7e0ceJcfQh6blOdPRxMfDeJPcD1wNvS/KFHvNJkrYwt6ir6lNVdV5V7QPeD3yjqj649GQ72Npaf2u99a39rdWnPv+P0m7n86gH0OfM9vbb+1urT86lpf6sLHJwVd0O3L6UJJKkDXlGLUmNs6glqXEWtSQ1zqKWpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1JjbOoJalxFrUkNc6ilqTGWdSS1DiLWpIaZ1FLUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxs0t6iSnJfluku8luSfJ1dsRTN2427e083U5o/4d8Laqeg1wAfCuJBctNZU6u9ovm9KOtzLvgKoq4InZ1efP3mqZoSRJz+o0o05ySpLDwFHg61V1cINj9icZJxlPJpOeY+pYoxEk0zd49rJjEGlnyvSEuePByRnAzcDHqurIZsetrq7WeDw+8XSaK4EF3oWSGpXkUFWtbnTbQs/6qKpHgduAd/WQS5LUQZdnfZw9O5MmyenAO4D7lpxLHa2tDZ1A0rLN/WEicA5wbZJTmBb7DVX11eXGUlfOpaWdr8uzPv4beO02ZJEkbcBXJkpS4yxqSWqcRS1JjbOoJalxFrUkNc6ilqTGWdSS1DiLWpIaZ1FLUuMsaklqnEUtSY2zqCWpcRa1JDXOopakxlnUktQ4i1qSGmdRS1LjLGpJapxFLUmN67IL+UuT3Jbk+0nuSXLlMgO1ullrq7kk7Xypqq0PSM4BzqmqO5O8ADgE/EVVfX+z+6yurtZ4PD6+QIE5kQbRai5JO0OSQ1W1utFtc8+oq+rnVXXn7PL/AvcC5/YbUZK0mYVm1En2Aa8FDm5w2/4k4yTjyWSyUIjRaHrGmqyvNX0betzQai5Ju8vc0cczByZ/BHwT+HRVfWWrYx19SNJiTmj0MVvg+cBNwBfnlbQkqV9dnvUR4Brg3qr6l2UHWltb9r9wfFrNJWnn6/KsjzcDdwB3A3+Y/fU/VtXXNrvPiYw+JGk32mr0sTLvzlX1bSC9p5IkdeIrEyWpcRa1JDXOopakxlnUktS4zi94WWjRZAI8cJx3Pwv4VY9x+mKuxZhrMeZazE7M9adVdfZGNyylqE9EkvFmT1EZkrkWY67FmGsxuy2Xow9JapxFLUmNa7Go/33oAJsw12LMtRhzLWZX5WpuRi1J+v9aPKOWJB3DopakxjVT1En+I8nRJEeGzrJuuzf27SrJaUm+m+R7s1xXD53pWElOSXJXkq8OneVYSe5PcneSw0ma+fWOSc5IcmOS+5Lcm+SNDWQ6f/Y4rb89nuSqoXMBJPn47OP+SJLrkpw2dCaAJFfOMt3T92PVzIw6ySXAE8B/VtWrh84Dx7ex7zblCrC3qp6YberwbeDKqvrOkLnWJfkEsAq8sKquGDrPuiT3A6tV1dQLJZJcC9xRVQeSnArsqapHB471jCSnAA8Db6iq430hW19ZzmX68f6qqvptkhuAr1XV5wfO9WrgeuBC4EngFuDDVfXDPtZv5oy6qr4F/M/QOY7V6sa+NfXE7OrzZ29NfMVNch7wHuDA0FlOBkleBFzCdHMOqurJlkp65jLgR0OX9DFWgNOTrAB7gJ8NnAfglcDBqvpNVT3NdNvC9/W1eDNF3bqtNvYdwmy8cBg4Cny9qprIBXwW+CTPbjLRkgJuTXIoyf6hw8y8HJgAn5uNiw4k2Tt0qOd4P3Dd0CEAquph4DPAg8DPgceq6tZhUwFwBHhLkjOT7AEuB17a1+IWdQezjX1vAq6qqseHzgNQVb+vqguA84ALZ996DSrJFcDRqjo0dJZNvLmqXge8G/jobNw2tBXgdcC/VdVrgV8D/zBspGfNRjHvBb48dBaAJC8G/pzpF7g/AfYm+eCwqaCq7gX+GbiV6djjMPD7vta3qOdofWPf2bfJtwHvGjgKwMXAe2ez4OuBtyX5wrCRnjU7G6OqjgI3M50nDu0h4KFjviO6kWlxt+LdwJ1V9cuhg8y8HfhJVU2q6ingK8CbBs4EQFVdU1Wvr6pLgEeAH/S1tkW9he3e2LerJGcnOWN2+XTgHcB9g4YCqupTVXVeVe1j+u3yN6pq8LMdgCR7Zz8QZjZaeCfTb1cHVVW/AH6a5PzZX10GDPrD6uf4AI2MPWYeBC5Ksmf2+XkZ058dDS7JS2Z/vozpfPpLfa09d8/E7ZLkOuBS4KwkDwFrVXXNsKm4GPgr4O7ZPBjmbOy7Tc4Brp39NP55wA1V1dRT4Rr0x8DN089tVoAvVdUtw0Z6xseAL87GDD8GPjRwHuCZL2jvAP5u6CzrqupgkhuBO4Gngbto5+XkNyU5E3gK+GifPxRu5ul5kqSNOfqQpMZZ1JLUOItakhpnUUtS4yxqSWqcRS1JjbOoJalx/wfElKm+CSUnQgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Test Data\n",
    "def load_regression_data():\n",
    "    x = np.array([1.0,2.0,3.0,2.5,3.0,3.0,4.0,4.0,4.5,5.0,5.0,6.0,6.0,7.0,8.0,8.0,9.0,9.0])\n",
    "    N = x.size\n",
    "    x = x.reshape(N,1)\n",
    "    y = np.array([2.0,2.0,4.0,4.0,3.0,4.0,4.0,5.0,6.0,5.0,6.0,7.0,9.0,8.0,7.0,8.0,8.0,9.0]).reshape(N,1)\n",
    "    return x,y\n",
    "\n",
    "x,y = load_regression_data()\n",
    "plt.plot(x,y,'b+')\n",
    "\n",
    "def design_matrix(x):\n",
    "    X = np.stack((x**0, x), axis=1)\n",
    "    X = X.squeeze()\n",
    "    return X\n",
    "    \n",
    "X = design_matrix(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:blue\">EXERCISE 2</span> \n",
    "\n",
    "Implement the cost function and its gradient for a general linear regression problem with design matrix $X$ and target value vector $y$.\n",
    "\n",
    "As cost function we use the average square distance between target values $y^{(i)}$ and values predicted by the model $\\hat{y}^{(i)}$. \n",
    "\n",
    "Don't get confused about the notation and what needs to get optimized, i.e. the linear regression parameters $\\alpha$. The data points $x,y$ and the design matrix are considered as additional parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 404,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lin_regression_cost(alpha, X, y):\n",
    "    \"\"\"\n",
    "    Cost function for the linear regression problem with affine linear model.\n",
    "\n",
    "    Arguments:    \n",
    "    alpha -- array of parameters to be optimized: np-array of shape (m,1)\n",
    "    X -- design matrix, array of shape (N, m)\n",
    "    y -- regression result, array of shape (N,1)\n",
    "    \n",
    "    Returns:\n",
    "    cost -- the cost function for the given parameter array\n",
    "    \"\"\"\n",
    "    # START YOUR CODE\n",
    "\n",
    "    yhat = np.dot(X,alpha)\n",
    "    \n",
    "    return (np.square(np.linalg.norm(y-yhat))/(2*(len(y))))\n",
    "\n",
    "    \n",
    "    # END YOUR CODE\n",
    "    \n",
    "def lin_regression_gradient_cost(alpha, X, y):\n",
    "    \"\"\"\n",
    "    Gradient of the cost function for the linear regression problem with affine linear model.\n",
    "    Arguments:\n",
    "    alpha -- array of parameters to be optimized: np-array of shape (m,1)\n",
    "    X -- design matrix, array of shape (N, m)\n",
    "    y -- regression result, array of shape (N,1)\n",
    "    \n",
    "    Returns:\n",
    "    gradient_cost -- the gradient of the cost function for the given parameter array\n",
    "    \"\"\"\n",
    "    # START YOUR CODE    \n",
    "    x = X[0]\n",
    "    yhat = np.dot(X,alpha)\n",
    "    XT = np.transpose(X)\n",
    "    X2T = 2 * XT\n",
    "    return ((np.dot(X2T, y) - np.dot(X2T,yhat))/(-2*(len(y))))\n",
    "    \n",
    "    \n",
    "    # END YOUR CODE\n",
    "    \n",
    "def optimized_parameters_exact(X,y):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    X -- design matrix, array of shape (N, m)\n",
    "    y -- regression result, array of shape (N,1)\n",
    "\n",
    "    Returns:\n",
    "    alpha_opt -- the optimized parameters by solving the normal equations\n",
    "    \"\"\"\n",
    "    # START YOUR CODE\n",
    "    XT = np.transpose(X)\n",
    "    XTy = np.dot(XT, y)\n",
    "    XTX = np.dot(XT, X)\n",
    "    return np.dot(np.linalg.inv(XTX), XTy)\n",
    "    \n",
    "    # END YOUR CODE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:green\">TEST</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {},
   "outputs": [],
   "source": [
    "########\n",
    "# TEST\n",
    "########\n",
    "x,y = load_regression_data()\n",
    "X = design_matrix(x)\n",
    "alpha0 = np.array([1,1]).reshape(2,1)\n",
    "np.testing.assert_almost_equal(0.5138888888888888, lin_regression_cost(alpha0, X=X, y=y))\n",
    "np.testing.assert_array_equal((2,1), lin_regression_gradient_cost(alpha0, X=X, y=y).shape)\n",
    "np.testing.assert_array_almost_equal(np.array([0.38888889,2.75]).reshape(2,1), lin_regression_gradient_cost(alpha0, X=X, y=y), decimal=8)\n",
    "alpha, _, _ = vanilla_gradient_descent(lin_regression_cost, lin_regression_gradient_cost, 0.001, alpha0, 200000, eps=1.0e-6, X=X, y=y)\n",
    "np.testing.assert_array_equal((2,1), alpha.shape)\n",
    "np.testing.assert_array_almost_equal(optimized_parameters_exact(X, y), alpha, decimal=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:blue\">EXERCISE 3 </span>\n",
    "\n",
    "Apply Vanilla Gradient Descent to the linear regression use case. \n",
    "Examine the number of iterations as a function of the learning rate: Choose the fixed starting value $(1,1)$ and an accuracy $\\epsilon = 1.0e^{-6}$.\n",
    "1. Determine the range $[\\rho_{\\rm min}, \\rho_{\\rm max}]$ from which the learning rate can be chosen at which Vanilla Gradient Descent converges. \n",
    "2. Determine the optimal learning rate $\\rho_{\\rm opt}$ at which the number of iterations becomes minimal.\n",
    "3. What happens at learning rates larger than $\\rho_{\\rm opt}$? Explain - also by studying the learning speed curve.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pmin:\n",
      "[[0.99281659]\n",
      " [0.94388669]]\n",
      "32524\n",
      "pmax:\n",
      "[[1.32539691]\n",
      " [0.85714335]]\n",
      "2209\n",
      "popt:\n",
      "[[1.32531441]\n",
      " [0.85715624]]\n",
      "734\n"
     ]
    }
   ],
   "source": [
    "# EXERCISE 3.1\n",
    "\n",
    "x,y = load_regression_data()\n",
    "X = design_matrix(x)\n",
    "\n",
    "# START YOUR CODE\n",
    "init = np.array([1,1]).reshape(2,1)\n",
    "\n",
    "xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(lin_regression_cost, lin_regression_gradient_cost, 1.0e-6 , init, 1.0e6, eps=1.0e-6, X=X, y=y)\n",
    "print(\"pmin:\")\n",
    "print(xopt)\n",
    "print(len(cost_hist))\n",
    "\n",
    "xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(lin_regression_cost, lin_regression_gradient_cost, 0.0634 , init, 1.0e6, eps=1.0e-6, X=X, y=y)\n",
    "print(\"pmax:\")\n",
    "print(xopt)\n",
    "print(len(cost_hist))\n",
    "\n",
    "xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(lin_regression_cost, lin_regression_gradient_cost, 0.063 , init, 1.0e6, eps=1.0e-6, X=X, y=y)\n",
    "print(\"popt:\")\n",
    "print(xopt)\n",
    "print(len(cost_hist))\n",
    "\n",
    "\n",
    "\n",
    "# END YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comments**:\n",
    "* $\\rho_{\\rm max} = 0.0634$ with $2209$ iterations. It converges at the point $xopt = (1.32539691, 0.85714335)$\n",
    "* $\\rho_{\\rm min} = 1.0e^{-6}$ with $32524$ iterations. It converges at the point $xopt = (0.99281659, 0.94388669)$\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3.2:**\n",
    "\n",
    "Answer:\n",
    "\n",
    "The optimal learning speed should ideally converge with a minimal amount of iterations. To achieve that, it should be at xopt with minimal iterations.\n",
    "\n",
    "With the optimal learning speed $\\rho_{\\rm opt} = 0.063 $, the VDG will converge after $734$ iterations at the point $(1.32531441, 0.85715624)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3.3:**\n",
    "\n",
    "Answer:\n",
    "\n",
    "With a learning rate larger than $\\rho_{\\rm opt}$, it will still converge but only if it does not get larger than $\\rho_{\\rm max}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Application to Rosenbrock Function\n",
    "\n",
    "Rosenbrock function is defined by\n",
    "\n",
    "$\\qquad f(x,y)= \\frac{1}{2}\\left((4(x^2-y))^2+(y-1)^2\\right)$\n",
    "\n",
    "\n",
    "#### <span style=\"color:blue\">EXERCISE 4</span>\n",
    "\n",
    "Implement Vanilla Gradient Descent applied to Rosenbrock's function: Implement the function and its gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rosenbrock(x):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    x -- a 2d np-array (of shape (2,1)) \n",
    "\n",
    "    Returns:\n",
    "    c -- value of the rosenbrock function\n",
    "    \"\"\"    \n",
    "    # START YOUR CODE  \n",
    "\n",
    "    X = x[0]\n",
    "    Y = x[1]\n",
    "    return 0.5 * (16 * (Y - X**2)**2 + (Y - 1)**2)\n",
    "    \n",
    "    # END YOUR CODE\n",
    "    \n",
    "def gradient_rosenbrock(x):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    x -- a 2d np-array (of shape (2,1)) \n",
    "\n",
    "    Returns:\n",
    "    gradient -- gradient of the rosenbrock function, np-array of shape (2,1)\n",
    "    \"\"\"    \n",
    "    # START YOUR CODE\n",
    "\n",
    "    X = x[0]\n",
    "    Y = x[1]\n",
    "    \n",
    "    gdX = -16 * (Y - X**2) * 2 * X\n",
    "    gdY = 16 * (Y - X**2) + (Y - 1)\n",
    "    \n",
    "    return np.array([gdX, gdY]).reshape(2, 1)\n",
    "    \n",
    "    # END YOUR CODE            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:green\">TEST</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "########\n",
    "# TEST\n",
    "########\n",
    "np.testing.assert_almost_equal(0.0, rosenbrock(np.array([1,1]).reshape(2,1)), decimal=8)\n",
    "np.testing.assert_almost_equal(0.0, rosenbrock(np.array([-1,1]).reshape(2,1)), decimal=8)\n",
    "np.testing.assert_almost_equal(17.0/2.0, rosenbrock(np.array([1,0]).reshape(2,1)), decimal=8)\n",
    "\n",
    "np.testing.assert_almost_equal(np.array([0,0]).reshape(2,1), gradient_rosenbrock(np.array([1,1]).reshape(2,1)), decimal=8)\n",
    "np.testing.assert_almost_equal(np.array([32,-17]).reshape(2,1), gradient_rosenbrock(np.array([1,0]).reshape(2,1)), decimal=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:blue\">AUFGABE 5</span>\n",
    "\n",
    "Apply Vanilla Gradient Descent as implemented in EXERCISE 1. to Rosenbrock's function.<br>\n",
    "\n",
    "1. Analyse the behavior of GD with different learning rates. Select a fixed initial value $(0.5,0.0)$ and a precision $1.0e^{-6}$. For comparison, prepare a plot with number of iterations needed vs learning rate. What learning rates are possible? Is there an optimal learning rate? **Remark:** Only accept solutions that lead to the minimum in the right half plane (located at $(1,1)$).  \n",
    "\n",
    "2. Analyse the behavior of GD with different learning rates - now selecting $(1.0,0.0)$ as initial value (und again a precision $1.0e^{-6}$). Is there a difference in the behavior? \n",
    "\n",
    "3. Analyse the behavior of GD with varying initial value at a fixed learning rate $\\rho=0.02$. Choose initial values always on the x-axis, i.e. $(x,0)$. Prepare a plot with the number of iterations vs initial values. \n",
    "\n",
    "4. Are there areas in the (x,y)-plane that are particularly problematic to be used for the starting values? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 488,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeE0lEQVR4nO3deXRc5Z3m8e9PtaiqtG+2bEnGNjaLMQGMhsVk6YGwp9tsndDNJCRDD9NJTjrbnJ6QZeiTpOd0ZjJJZ4MOCZ0m6SRAWIITSDoOS9PBwYkMtsE2Btl4kxfZsiRrL1XpnT/qSpaNbJcsla5U9/mcU+feeu/C+57Cz3v11ntvmXMOEREJhgK/KyAiIlNHoS8iEiAKfRGRAFHoi4gEiEJfRCRAwn5X4ESqq6vd/Pnz/a6GiMiMsnbt2oPOuZqxtk3r0J8/fz5NTU1+V0NEZEYxsx3H26bhHRGRAFHoi4gEiEJfRCRAFPoiIgGi0BcRCRCFvohIgCj0RUQCJC9Dv6Wjj6/9Zgs72nr8roqIyLSSl6F/uG+Qbz7TzKsth/2uiojItJKXod9QmQBgV3uvzzUREZle8jL0iwvDVBZF2XlIoS8iMlpehj5krvZ3KfRFRI6St6E/rzKhK30RkWPkcejHaWnvIz2kH34XERmWt6HfUJEgNeTY29nnd1VERKaNvA39ed4MHg3xiIgckbehPzJtU6EvIjIib0N/TlmMUIGx65CGd0REhuVt6IdDBdSVxzW8IyIySt6GPmjapojIsfI69HWDlojI0fI89OO09STpGUj5XRURkWkhr0N/nh68JiJylECE/s42hb6ICAQl9DWuLyICZBn6ZvZJM9toZq+a2U/NLGZmC8xsjZk1m9lDZhb19i303jd72+ePOs9dXvkWM7s6R20aURaPUFIYZne75uqLiEAWoW9mdcDfAI3OuaVACLgV+ArwdefcIqAduMM75A6g3Sv/urcfZrbEO+4c4BrgHjMLTW5z3lJ3GjRtU0RkRLbDO2EgbmZhIAHsBS4HHvG2PwDc4K2v8N7jbb/CzMwrf9A5N+CcexNoBi6acAtOQnP1RUSOOGnoO+dagK8CO8mEfSewFuhwzg3PhdwN1HnrdcAu79iUt3/V6PIxjhlhZneaWZOZNR04cOBU2nSUeVWZufrO6RHLIiLZDO9UkLlKXwDMBYrIDM/khHPuPudco3OusaamZsLna6iIM5Aa4kDXwCTUTkRkZstmeOfdwJvOuQPOuUHgMeAyoNwb7gGoB1q89RagAcDbXga0jS4f45icadAMHhGREdmE/k7gEjNLeGPzVwCbgGeBW7x9bgee8NZXeu/xtj/jMmMrK4Fbvdk9C4DFwB8mpxnHp2mbIiJHhE+2g3NujZk9ArwEpICXgfuAJ4EHzezLXtn93iH3Az8ys2bgEJkZOzjnNprZw2Q6jBTwUedcepLb8xZ1FXHMFPoiIpBF6AM45+4G7j6meBtjzL5xzvUDf36c8/w98PfjrOOEFIZD1JbG9Fx9ERHy/I7cYXrapohIRiBCX3P1RUQyAhP6+w730z+Y868QRESmtUCEfkNlHICWDo3ri0iwBSL0NW1TRCQjEKE/fIOWvswVkaALROjXFBcSixTox1REJPACEfpmRkNFQj+bKCKBF4jQh+Fpm/oiV0SCLTChP3yDlh6xLCJBFpjQn1eZoHsgRXvvoN9VERHxTWBCXzN4REQCFPqaqy8iEqDQH74rV6EvIkEWmNBPRMNUF0c1vCMigRaY0AdvBo/m6otIgAUq9PWIZREJusCF/p6OfgbTQ35XRUTEF4EK/YaKBOkhx96Ofr+rIiLii2CF/vBcfY3ri0hABSr051Vprr6IBFugQr+2NEYkZAp9EQmsQIV+qMCoK48r9EUksAIV+pAZ19+t0BeRgApc6GuuvogEWSBDv713kMP9esSyiARP4EJfj1gWkSALXOjPGwl9/XSiiARP4EJfV/oiEmSBC/2yeISyeERf5opIIAUu9CHzgyoKfREJokCG/jw9V19EAiqQoZ+5QauPoSHnd1VERKZUIEN/XmWCZHqI/V16xLKIBEsgQ7+hwnvaZpuGeEQkWAIZ+iNz9ds1V19EgiWQoT+3PE6B6bn6IhI8gQz9aLiAOWVx3aAlIoGTVeibWbmZPWJmr5nZZjO71MwqzWyVmb3hLSu8fc3MvmlmzWa2wcyWjTrP7d7+b5jZ7blqVDY0V19EgijbK/1vAL92zp0FnAdsBj4DPO2cWww87b0HuBZY7L3uBO4FMLNK4G7gYuAi4O7hjsIP8yoTutIXkcA5aeibWRnwTuB+AOdc0jnXAawAHvB2ewC4wVtfAfzQZbwIlJvZHOBqYJVz7pBzrh1YBVwziW0Zl3mVCVq7BuhLpv2qgojIlMvmSn8BcAD4gZm9bGbfN7MiYLZzbq+3zz5gtrdeB+wadfxur+x45UcxszvNrMnMmg4cODC+1ozD8IPXduvOXBEJkGxCPwwsA+51zl0A9HBkKAcA55wDJuX2Vufcfc65RudcY01NzWScckzDoa9xfREJkmxCfzew2zm3xnv/CJlOYL83bIO3bPW2twANo46v98qOV+6LeXrEsogE0ElD3zm3D9hlZmd6RVcAm4CVwPAMnNuBJ7z1lcAHvFk8lwCd3jDQvwFXmVmF9wXuVV6ZL6qKoiSiIXbqx1REJEDCWe73MeDHZhYFtgEfItNhPGxmdwA7gPd6+z4FXAc0A73evjjnDpnZl4A/evt90Tl3aFJacQrMTD+SLiKBk1XoO+fWAY1jbLpijH0d8NHjnOefgX8eR/1yqr5C0zZFJFgCeUfusOHn6mf6KRGR/Bfw0I/Tm0zT1pP0uyoiIlMi0KGvaZsiEjSBDn1N2xSRoAl06NdXKPRFJFgCHfrxaIhZJYUa3hGRwAh06ENmXF+hLyJBEfjQzzxiWXflikgwBD70GyoT7O3sI5ka8rsqIiI5F/jQn1eZYMjBng5d7YtI/gt86DdUxAHN1ReRYAh86M+r0g1aIhIcgQ/92SUxoqECdukXtEQkAAIf+gUFRn1lXDdoiUggBD70ARoqNFdfRIJBoU9mBs/ONoW+iOQ/hT6Z0D/cn6Kzd9DvqoiI5JRCnyOPWNaXuSKS7xT6QEOl5uqLSDAo9NGPqYhIcCj0gdJYhIpERNM2RSTvKfQ98/SIZREJAIW+p74yoSt9Ecl7Cn3P6dVF7Grvo70n6XdVRERyRqHvuWbpHNJDjl9u2ON3VUREckah71kyt5Szakt45KUWv6siIpIzCv1RbrmwnvW7Omhu7fa7KiIiOaHQH2XF+XWECoxHX9rtd1VERHJCoT9KTUkh7zqjhsdfaiE95PyujojIpFPoH+PmZfXsO9zP77e2+V0VEZFJp9A/xhVnz6I0FtYQj4jkJYX+MWKREO85by6/fnUf3QMpv6sjIjKpFPpjuHlZPX2DaZ56Za/fVRERmVQK/TEsm1fOguoiHl2rIR4RyS8K/TGYGTcvq2PNm4f0PB4RySsK/eO4cVk9AI+/rDt0RSR/KPSPo648zqULq3jspd04pzn7IpIfFPoncPOF9Wxv62Xtjna/qyIiMikU+idw7dJaEtGQ5uyLSN7IOvTNLGRmL5vZL733C8xsjZk1m9lDZhb1ygu9983e9vmjznGXV77FzK6e9NZMsqLCMNcsreWX6/fSP5j2uzoiIhM2niv9jwObR73/CvB159wioB24wyu/A2j3yr/u7YeZLQFuBc4BrgHuMbPQxKqfe7csq6drIMVvNu33uyoiIhOWVeibWT1wPfB9770BlwOPeLs8ANzgra/w3uNtv8LbfwXwoHNuwDn3JtAMXDQJbcipSxZWMbcsxmMa4hGRPJDtlf4/An8LDHnvq4AO59zwcwp2A3Xeeh2wC8Db3untP1I+xjEjzOxOM2sys6YDBw5k35IcKSgwblxWx/OvH6D1cL/f1RERmZCThr6ZvQdodc6tnYL64Jy7zznX6JxrrKmpmYr/5EndtKyeIQc/X6c5+yIys2VzpX8Z8Gdmth14kMywzjeAcjMLe/vUA8OJ2AI0AHjby4C20eVjHDOtnV5TzAXzynl0bYvm7IvIjHbS0HfO3eWcq3fOzSfzRewzzrnbgGeBW7zdbgee8NZXeu/xtj/jMkm5ErjVm92zAFgM/GHSWpJjNy+rZ8v+LjbuOex3VURETtlE5un/T+BTZtZMZsz+fq/8fqDKK/8U8BkA59xG4GFgE/Br4KPOuRkzD/I9b5tDNFSgOfsiMqPZdB6uaGxsdE1NTX5XY8RHfryWNdsO8eJnryAS0n1tIjI9mdla51zjWNuUXONw87J62nqSPLfF/1lFIiKnQqE/Du88o4bq4qiesy8iM5ZCfxwioQJWnF/H06/tp70n6Xd1RETGTaE/Tjctq2Mw7fjFhj1+V0VEZNwU+uN0ztwyzqot4dGXZsQtBiIiR1Hon4JbLqxn/a4Omlu7/a6KiMi4KPRPwYrz6wgVmObsi8iMo9A/BTUlhbzrjBoef6mF9ND0vc9BRORYCv1TdNOyOvYd7mf11oN+V0VEJGsK/VP07rNnUxoL8/3/eFMPYRORGUOhf4pikRCfvPIM/v31A/x4zU6/qyMikhWF/gR8cPl83nVGDV9+chPNrV1+V0dE5KQU+hNgZvzfP38biWiYv/npOgZSM+ahoSISUAr9CZpVEuP/3Pw2Nu09zNd+87rf1REROSGF/iR495LZ3HbxPL77/DZeaNZsHhGZvhT6k+Tz1y9hYU0Rn354vR7GJiLTlkJ/ksSjIb556wW09Qzw2cdf0TROEZmWFPqTaGldGf/jqjP51av7+FmTHtEgItOPQn+S/bd3LGT56VX83S828ubBHr+rIyJyFIX+JCsoMP7fe88jEirgEw++zGB6yO8qiYiMUOjnwJyyOP9w07ms393JN377ht/VEREZodDPkWvPncN7G+v5znPNrNnW5nd1REQAhX5O3f2n53BaZYJPPbyezr5Bv6sjIqLQz6WiwjD/eOsF7Dvcz+d//qqmcYqI7xT6OXZ+QzmffPdifrF+Dz9fp9/VFRF/KfSnwIf/ZBH/aX4FX/j5RnYd6vW7OiISYAr9KRAqML723vMx4BMPrSOlaZwi4hOF/hRpqEzw5RuXsnZHOx/76cv0D+oxzCIy9RT6U2jF+XV84T1L+PXGffzl916krXvA7yqJSMAo9KfYHW9fwL23LWPjnsPcdO9qPapBRKaUQt8H1yydw0/vvISu/hQ33fMCa3cc8rtKIhIQCn2fLJtXwWMfXk55IspffG8NT27Y63eVRCQAFPo+ml9dxKMfXs7b6sr46E9e4r7nt+oGLhHJKYW+zyqLovzrX13M9efO4X8/9Rr/64mNmtIpIjkT9rsCArFIiG/9xQXUV8T57vPb2NPRx7f+8gISUX08IjK5dKU/TRQUGHdddzZfWnEOz25p5X3ffZHWrn6/qyUieUahP828/9L5fO8DjTS3dnPjd1bT3Nrld5VEJI8o9KehK86ezcP//VKS6SFuumc1v9+q5/GLyOQ4aeibWYOZPWtmm8xso5l93CuvNLNVZvaGt6zwys3MvmlmzWa2wcyWjTrX7d7+b5jZ7blr1sx3bn0Zj39kObNKY9z2/Rf5u5Ub9Ux+EZmwbK70U8CnnXNLgEuAj5rZEuAzwNPOucXA0957gGuBxd7rTuBeyHQSwN3AxcBFwN3DHYWMrb4iwWMfWc5/ueQ0fvj77Vz+1ed4+I+7GBrStE4ROTUnDX3n3F7n3EveehewGagDVgAPeLs9ANzgra8AfugyXgTKzWwOcDWwyjl3yDnXDqwCrpnMxuSj0liEL65Yyi8+9nYWVBfxt49u4MZ7V7Nhd4ffVRORGWhcY/pmNh+4AFgDzHbODd9Gug+Y7a3XAbtGHbbbKzte+bH/jTvNrMnMmg4cODCe6uW1c+aW8bO/vpSvv+889nT0seI7L3DXYxs41JP0u2oiMoNkHfpmVgw8CnzCOXd49DaXuY10UsYcnHP3OecanXONNTU1k3HKvGFm3HhBPc98+l381dsX8LOm3fznrz7Hj36/nbSGfEQkC1mFvplFyAT+j51zj3nF+71hG7xlq1feAjSMOrzeKzteuYxTSSzC565fwq8+/g6W1pXyhSc28qff+h1N2/XgNhE5sWxm7xhwP7DZOfe1UZtWAsMzcG4HnhhV/gFvFs8lQKc3DPRvwFVmVuF9gXuVVyanaPHsEv71jou557ZldPQmueWffs+nHlpH62Hd1CUiY8vmPv/LgPcDr5jZOq/ss8A/AA+b2R3ADuC93rangOuAZqAX+BCAc+6QmX0J+KO33xedc7o0nSAz47pz5/AnZ9Zwz7Nbue/5bfxm037ufOdCbrt4HlXFhX5XUUSmEZvOT3VsbGx0TU1NfldjRtl+sIcvP7mZ327eTzRcwA3nz+WDyxewZG6p31UTkSliZmudc41jblPo56fm1i7+ZfV2Hl3bQt9gmosXVPKhyxZw5ZLZhArM7+qJSA4p9AOss3eQh5p28sDqHbR09FFXHuf25afxvsZ5lCUifldPRHJAoS+k0kP8dvN+fvDCdta8eYh4JMTNF9bxweXzWTSrxO/qicgkUujLUTbu6eRfXtjOE+v3kEwN8Y7F1fzXyxbwzjNqNPQjkgcU+jKmtu4BfrJmJz96cQetXQNUF0e5+pxarj93DhctqCQc0kNYRWYihb6cUDKVGfp5csNennmtlb7BNFVFUa46p5brzq3lkoVVRNQBiMwYCn3JWl8yzXNbWnnq1X08vXk/vck0FYkIVy2p5dpza7lsUbU6AJFpTqEvp6R/MM2/v36Ap17Zy9ObW+keSFEWj3Dlktlcf+4cLltUTTSsDkBkulHoy4T1D6b5jzcO8qtX9rJq0366BlIUF4a5eEElly2q5rJF1Zwxu5jMUztExE8nCv1sHsMgQiwS4sols7lyyWwGUmleaD7Iqk2trN56kKdfyzxrr7q4kOWnV3HZoiqWn15NQ2XC51qLyLEU+jJuheEQl581m8vPyvyEwu72XlY3t/HC1oO80NzGyvV7ADitKsHy06u5bFEVly6s0nOARKYBDe/IpHLO8UZrN7974yCrtx7kxW2H6B5IAXD2nFIuXlDJ+Q3lnNdQzvyqhIaDRHJAY/rim1R6iA0tnaxuzvwVsG5XB32DaQBKY2HOayjnvPpyb1nGrNKYzzUWmfkU+jJtpNJDvNHazfpdHazf3cn6XR1s2d818stfc8piR3UC59aXURLTM4JExkNf5Mq0EQ4VcPacUs6eU8qtF2XK+pJpNu3tZN2uTq8z6ODXG/cBYAbzKhOcObuEs2pLOLO2lDNrS5hfldAdwyKnQKEvvotHQ1x4WiUXnlY5Utbek2RDSycbdnXw2r4uXtt3mN9u3s/wTwFHwwUsnlXMmbVHOoOzakuYVVKo7wlETkDDOzJj9A+maW7tZovXCby2r4st+7po7RoY2ac8EeGM2SUsrC5iQXUR86uLWFhdRENlglgk5GPtRaaOhnckL8QiIZbWlbG0ruyo8vaepNcBHGbL/i5e39/Nqk37aetJjuxjBnXl8UxHUJXpEIZf9RVxDRVJYCj0ZcarKIpy6elVXHp61VHlnX2DbD/Yw/a2HrYdyCzfPNjDz19uocubRgoQLjDqK+LUVyS8ZWa9oTKzrCkupECPnJY8odCXvFUWj2RmATWUH1XunKOtJ8n2gz1sO5jpCHYd6mVXex+/3byfg93Jo/aPhgqoG+kMjnQOc8vj1JbGmF0a0zOIZMZQ6EvgmBnVxYVUFxfSOL/yLdv7kmlaOjKdwO72Pna393rLPlZtemunAFBdHKW2LEZtaZw5ZTFqy2KZZWlmvbYsRiKqf27iP/1fKHKMeDTEolklx/0Zyd5kipb2PvZ29rOvs599h/u99UwH0bTjEB29g285rjQWZlZpjJriQmaVFh5ZlhRSUxwbKStPRDQDSXJGoS8yTolomMWzS1g8+/i/LdyXTHudQd9Ix7Cvs58DXQO0dg3w8s4OWrv66R8cesuxkZBRU5zpDKqLC6ksilJVXEhVUZTKoiiVxdGR9aqiQuJRzUqS7Cn0RXIgHg2NzA46Hucc3QOpkY7g6GWmg9jT2c+rezo51JNkMD329OpENOR1AJmOoKIoSkUiSkUiQnkiU1aeiHhlmXVNXw0uhb6IT8yMkliEkliEhTXFJ9zXOUfXQIq27iSHega8ZZK2nsxyeL21a4DX93fT3pukN5k+7vkS0dBIB1CRiFIWj1CWiGSW3qvcW5bGI5R724oLwxp6muEU+iIzgJlRGotQGouc8K+H0foH03T0DtLem6S9Nzmy3tE7SHtPkkOjyvZ09nG4b5DOvsHj/kUBECowSmNhyuKZzqo0Hqak0FvGIpTEwpQOL+NH3pfGIhTHwhQVhigM668MPyn0RfJULBKitixEbVn2Ty51ztGbTNPpdQCdfYN09A6OdAidfYN09CXp7EvR1T9IV3+KA13dHPbe95zgr4th0VABRYUhigrDFA+/YmGKCsOUFGaWR6+HKIqGR/YvKgx5yzCJaEh/eYyTQl9ERpjZSOjOLY+P+/hUeojugRRd/Sk6+zKdQlf/IIf7U3R7nUL3QIru/hQ9Aym6BjLL9p4kOw/10jO8LYvOI1NfvA7hSCcSj2TW49EQRdEQiWimc0iMXi8Mk4iESBRmyuKRzPaYt4xHQnl7Q55CX0QmTThUQHkiSnkiSsMEzjM05OhJpugZyHQSPd6reyBFTzJF90D66LKBzL49yRS9yTStXf30JtP0DqTp9cpSQ+N7zlhhuGCkA4hHM69EJEwsGiIeKSAeyXQSw6/M+wLi0RCxcMjbL1MWi2TKCiMFFIYLKAyHKAxnyqPhAkJT2MEo9EVk2ikoOPIl92RJpoZGOoDhZc9Amr7BFH3JzLb+wTS9yTR9g2n6vOVR772hr9bDmbJ+r7x/cIhk+q3Tb7MVCdlRHUFhuIArzp7F565fMmntH6bQF5FAiIYLiIajlCdyc/70kMt0Al5nkHkNjXQYydQQA6kh+gfTDKSGGEh5y8Eh+lNpBgZHlaWGqC0b//BaNhT6IiKTIFRw5PuQ6UxPiRIRCRCFvohIgCj0RUQCRKEvIhIgCn0RkQBR6IuIBIhCX0QkQBT6IiIBYs6N73kUU8nMDgA7vLfVwEEfq+OXILZbbQ4GtTl3TnPO1Yy1YVqH/mhm1uSca/S7HlMtiO1Wm4NBbfaHhndERAJEoS8iEiAzKfTv87sCPgliu9XmYFCbfTBjxvRFRGTiZtKVvoiITJBCX0QkQHwLfTO7xsy2mFmzmX1mjO2FZvaQt32Nmc0fte0ur3yLmV2d7Tn9lqM2bzezV8xsnZk1TVFTsnaqbTazKjN71sy6zezbxxxzodfmZjP7pplNq1+wzlGbn/POuc57zZqi5mRtAu2+0szWep/pWjO7fNQx+fpZn6jNuf2snXNT/gJCwFZgIRAF1gNLjtnnI8A/eeu3Ag9560u8/QuBBd55Qtmc089XLtrsbdsOVPvdvhy0uQh4O/DXwLePOeYPwCWAAb8CrvW7rVPQ5ueARr/bl6N2XwDM9daXAi0B+KxP1OacftZ+XelfBDQ757Y555LAg8CKY/ZZATzgrT8CXOH18iuAB51zA865N4Fm73zZnNNPuWjzdHfKbXbO9Tjnfgf0j97ZzOYApc65F13mX8gPgRty2YhxmvQ2zxATaffLzrk9XvlGIO5dIefzZz1mm6ei0n6Ffh2wa9T73V7ZmPs451JAJ1B1gmOzOaefctFmAAf8xvsT8c4c1HsiJtLmE51z90nO6adctHnYD7w/978w3YY5mLx23wy85JwbIDif9eg2D8vZZz29f8FXsvF251yLN+63ysxec84973elZNLd5n3OJcCjwPvJXPnmDTM7B/gKcJXfdZkqx2lzTj9rv670W4CGUe/rvbIx9zGzMFAGtJ3g2GzO6adctBnn3PCyFXic6TXsM5E2n+ic9Sc5p59y0ebRn3MX8BOm1+cME2y3mdWT+f/3A865raP2z9vP+jhtzvln7Vfo/xFYbGYLzCxK5guOlcfssxK43Vu/BXjGG9dbCdzqjfktABaT+bInm3P6adLbbGZF3tUAZlZE5mrh1SloS7Ym0uYxOef2AofN7BLvz94PAE9MftVP2aS32czCZlbtrUeA9zC9PmeYQLvNrBx4EviMc+6F4Z3z+bM+Xpun5LOe6m+8R31DfR3wOplvvz/nlX0R+DNvPQb8jMyXln8AFo469nPecVsY9W3+WOecTq/JbjOZWQPrvdfGPGzzduAQ0E1mvHSJV95I5h/CVuDbeHeWT5fXZLeZzKyetcAG73P+Bt7sren0OtV2A58HeoB1o16z8vmzPl6bp+Kz1mMYREQCRHfkiogEiEJfRCRAFPoiIgGi0BcRCRCFvohIgCj0RUQCRKEvIhIg/x9kGBsTBZBk8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# EXERCISE 5.1\n",
    "\n",
    "# START YOUR CODE\n",
    "init = np.array([0.5,0.0]).reshape(2,1)\n",
    "it = []\n",
    "ls =  []\n",
    "\n",
    "for l in np.linspace(0.001, 0.025851, num=25):\n",
    "    xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(rosenbrock, gradient_rosenbrock, l, init, 1.0e6, eps=1.0e-6)\n",
    "    it = np.append(it, len(cost_hist))\n",
    "    ls = np.append(ls, l)\n",
    "\n",
    "plt.figure(3)\n",
    "plt.plot(ls, it)\n",
    "plt.show()\n",
    "\n",
    "# END YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comment**:\n",
    "The graph plots the number of iterations needed (vertical) vs learning rate (horizontal).\n",
    "\n",
    "In the graph we can observe that with a higher lerning rate, up to $0.025851$, the number of iterations needed is lowered.\n",
    "\n",
    "Here, the $\\rho_{\\rm opt}$ is the same as $\\rho_{\\rm max}$.\n",
    "\n",
    "At $\\rho_{\\rm opt} = 0.025851$ the number of iterations needed are just $499$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjd0lEQVR4nO3deXhd1Xnv8e8rHR1NtmVJlgckGRtwmCejggPcDpCYIWlMIeHSmyZuyr1+mpLepsPTQtNebkPTNrlNk3DzhNYN5JoODAFSu4QCLkOSQhlkMAYDxvKAJeFBtmRZ8/jeP/aSfGwk+1jW0ZG0f5/nOc9ee+2191mLY961tfbae5u7IyIi8ZCT7QqIiMjEUdAXEYkRBX0RkRhR0BcRiREFfRGRGElkuwLHMmfOHF+0aFG2qyEiMqVs2LBhv7tXjLRtUgf9RYsWUVtbm+1qiIhMKWb2/mjbNLwjIhIjCvoiIjGioC8iEiMK+iIiMaKgLyISIwr6IiIxoqAvIhIj0zLoNx7s4m+e3sL7BzqyXRURkUllWgb9Q1193P1sHW81Hsp2VUREJpVpGfSry4oA2NXcmeWaiIhMLtMy6M/IT1BenKS+RUFfRCTVtAz6AFVlRdTrTF9E5AjTNugvLCvS8I6IyFGmcdAvpLGli/6BwWxXRURk0kgr6JvZ75rZZjN7y8weMLMCM1tsZi+bWZ2ZPWRmyVA2P6zXhe2LUo5zR8jfYmbXZKhNQHSm3z/o7G7tzuTXiIhMKccN+mZWCfxPoMbdzwNygVuArwPfcvczgBbg1rDLrUBLyP9WKIeZnRP2Oxe4FviemeWOb3MOG5rBo4u5IiKHpTu8kwAKzSwBFAG7gauAR8L2NcANIb0irBO2X21mFvIfdPced98B1AGXnnQLRlFdGoK+xvVFRIYdN+i7eyPw18AuomDfCmwADrp7fyjWAFSGdCVQH/btD+XLU/NH2GeYma0ys1ozq21qahpLmwBYUFJAIsd0MVdEJEU6wzulRGfpi4FTgGKi4ZmMcPfV7l7j7jUVFSO+4jEtidwcKksL2dXcNY61ExGZ2tIZ3vkYsMPdm9y9D3gMuAKYHYZ7AKqAxpBuBKoBwvYS4EBq/gj7ZISmbYqIHCmdoL8LWGZmRWFs/mrgbeA54NOhzEpgbUivC+uE7c+6u4f8W8LsnsXAEuCV8WnGyKp1g5aIyBESxyvg7i+b2SPAa0A/8DqwGvgx8KCZ/XnIuzfsci/wD2ZWBzQTzdjB3Teb2cNEHUY/cJu7D4xze45QXVpEc0cv7T39zMg/blNFRKa9tCKhu98J3HlU9nZGmH3j7t3AZ0Y5zteAr51gHcdsYdnhGTxnL5g1UV8rIjJpTds7cuFw0Ne4vohIJBZBX+P6IiKRaR30S4rymFWQ0Jm+iEgwrYM+aAaPiEiqaR/0NVdfROSwWAT9+pYuBgc921UREcm6aR/0q8uK6O0fZF9bT7arIiKSddM+6GvapojIYdM+6Fdr2qaIyLBpH/QrZxdipjN9ERGIQdBPJnI4paRQZ/oiIsQg6ANUlxXqTF9EhJgEfc3VFxGJxCbo72vrobsvo09yFhGZ9GIR9Idm8DS06GxfROItVkFfQzwiEnfpvBj9TDPbmPI5ZGZfNrMyM1tvZlvDsjSUNzO728zqzGyTmS1NOdbKUH6rma0c/VvH1/ANWgcU9EUk3o4b9N19i7tf5O4XAZcAncCPgNuBZ9x9CfBMWAe4juj9t0uAVcA9AGZWRvT2rcuI3rh151BHkWnlxUmKkrnsau6aiK8TEZm0TnR452pgm7u/D6wA1oT8NcANIb0CuN8jLwGzzWwBcA2w3t2b3b0FWA9ce7INSIeZaQaPiAgnHvRvAR4I6Xnuvjuk9wDzQroSqE/ZpyHkjZY/IapKi3QhV0RiL+2gb2ZJ4FPAD4/e5u4OjMuzi81slZnVmlltU1PTeBwSODxXP6qqiEg8nciZ/nXAa+6+N6zvDcM2hOW+kN8IVKfsVxXyRss/gruvdvcad6+pqKg4geod28KyQjp7BzjQ0TtuxxQRmWpOJOj/KoeHdgDWAUMzcFYCa1PyPx9m8SwDWsMw0FPAcjMrDRdwl4e8CbGwXNM2RUTSCvpmVgx8HHgsJfuvgI+b2VbgY2Ed4AlgO1AH/D3wWwDu3gzcBbwaPl8NeRNioR6xLCJCIp1C7t4BlB+Vd4BoNs/RZR24bZTj3Afcd+LVPHlVpZqrLyISiztyAQrycpk7M596zeARkRiLTdAHPW1TRCR2Qb9ed+WKSIzFKuhXlxXxQWsXvf2D2a6KiEhWxC7ou0PjQZ3ti0g8xSroa9qmiMRdLIO+LuaKSFzFKujPnZlPMpGjM30Ria1YBf2cHKO6tFBn+iISW7EK+hBdzFXQF5G4il3Q1w1aIhJnsQz6bd39tHb2ZbsqIiITLnZBv1ozeEQkxmIX9DVtU0TiLHZBX2f6IhJnsQv6M/ITlBUnFfRFJJZiF/QhOttv0HP1RSSG0n1d4mwze8TM3jWzd8zso2ZWZmbrzWxrWJaGsmZmd5tZnZltMrOlKcdZGcpvNbOVo39jZmnapojEVbpn+t8BnnT3s4ALgXeA24Fn3H0J8ExYB7gOWBI+q4B7AMysDLgTuAy4FLhzqKOYaAvLCmls6aJ/QI9YFpF4OW7QN7MS4OeBewHcvdfdDwIrgDWh2BrghpBeAdzvkZeA2Wa2ALgGWO/uze7eAqwHrh3HtqRtYVkR/YPO7tbubHy9iEjWpHOmvxhoAn5gZq+b2ffNrBiY5+67Q5k9wLyQrgTqU/ZvCHmj5R/BzFaZWa2Z1TY1NZ1Ya9JUXapHLItIPKUT9BPAUuAed78Y6ODwUA4A7u6Aj0eF3H21u9e4e01FRcV4HPJDNG1TROIqnaDfADS4+8th/RGiTmBvGLYhLPeF7Y1Adcr+VSFvtPwJt6CkgESOUa8ZPCISM8cN+u6+B6g3szND1tXA28A6YGgGzkpgbUivAz4fZvEsA1rDMNBTwHIzKw0XcJeHvAmXyM2hsrSQXXpJuojETCLNcr8N/JOZJYHtwBeIOoyHzexW4H3g5lD2CeB6oA7oDGVx92Yzuwt4NZT7qrs3j0srxkDTNkUkjtIK+u6+EagZYdPVI5R14LZRjnMfcN8J1C9jqkqLeGrznmxXQ0RkQsXyjlyIzvSbO3pp7+nPdlVERCZMrIM+aNqmiMRL7IO+xvVFJE5iH/R1pi8icRLboF9SlMfMgoTO9EUkVmIb9EHTNkUkfhT0FfRFJEZiH/QbWroYHByXxwaJiEx6sQ761WVF9PYPsq+tJ9tVERGZELEP+qBpmyISH7EO+pqrLyJxE+ugXzm7EDMFfRGJj1gH/WQih1NKCmlQ0BeRmIh10AeoLivUmb6IxEbsg77m6otInMQ+6FeXFrGvrYeu3oFsV0VEJOPSCvpmttPM3jSzjWZWG/LKzGy9mW0Ny9KQb2Z2t5nVmdkmM1uacpyVofxWM1s52vdNpIXl0QyeBr0vV0Ri4ETO9H/J3S9y96E3aN0OPOPuS4BnwjrAdcCS8FkF3ANRJwHcCVwGXArcOdRRZJPm6otInJzM8M4KYE1IrwFuSMm/3yMvAbPNbAFwDbDe3ZvdvQVYD1x7Et8/LvSIZRGJk3SDvgNPm9kGM1sV8ua5++6Q3gPMC+lKoD5l34aQN1r+EcxslZnVmlltU1NTmtUbu/LiJEXJXHY1d2X8u0REsi2tF6MDV7p7o5nNBdab2bupG93dzWxcnlrm7quB1QA1NTUZfxKamVFdqhk8IhIPaZ3pu3tjWO4DfkQ0Jr83DNsQlvtC8UagOmX3qpA3Wn7WVZcVaXhHRGLhuEHfzIrNbOZQGlgOvAWsA4Zm4KwE1ob0OuDzYRbPMqA1DAM9BSw3s9JwAXd5yMu6obn67nrEsohMb+kM78wDfmRmQ+X/2d2fNLNXgYfN7FbgfeDmUP4J4HqgDugEvgDg7s1mdhfwaij3VXdvHreWnISFZYV09Q1woKOXOTPys10dEZGMOW7Qd/ftwIUj5B8Arh4h34HbRjnWfcB9J17NzBqaq7+ruVNBX0SmtdjfkQvRXbmgaZsiMv0p6ANVIejvOqCgLyLTm4I+UJjMZe7MfE3bFJFpT0E/WFhWRL2evyMi05yCfrCwrIh63ZUrItOcgn7wkfkzaTzYpadtisi0pqAffPKCBQD86LVJcZOwiEhGKOgHVaVFLDutjMdeb9SduSIybSnop7hpaRU79nfw2q6D2a6KiEhGKOinuO78BRTm5fLoaw3ZroqISEYo6KeYkZ/g2vPm8/gbH9Ddp3fmisj0o6B/lJuWVnGou59n3tl3/MIiIlOMgv5RPnp6OfNnFWiIR0SmJQX9o+TmGL+ytJKfvNdEU1tPtqsjIjKuFPRHcNPSSgYGnbUbNWdfRKYXBf0RnDF3JhdWlfCYbtQSkWkm7aBvZrlm9rqZPR7WF5vZy2ZWZ2YPmVky5OeH9bqwfVHKMe4I+VvM7Jpxb804uumSKt7efYh3dh/KdlVERMbNiZzp/w7wTsr614FvufsZQAtwa8i/FWgJ+d8K5TCzc4BbgHOBa4HvmVnuyVU/c375glPIyzUe3aALuiIyfaQV9M2sCvgE8P2wbsBVwCOhyBrghpBeEdYJ268O5VcAD7p7j7vvIHqH7qXj0IaMKC1OctVZc/mXjR/QPzCY7eqIiIyLdM/0vw38ITAU/cqBg+7eH9YbgMqQrgTqAcL21lB+OH+EfYaZ2SozqzWz2qampvRbkgE3Lq1if3sPP9u6P6v1EBEZL8cN+mb2SWCfu2+YgPrg7qvdvcbdayoqKibiK0f1S2fOpbQoT3P2RWTaSOdM/wrgU2a2E3iQaFjnO8BsM0uEMlXA0FSXRqAaIGwvAQ6k5o+wz6SUTOSw4qJKnn57L61dfdmujojISTtu0Hf3O9y9yt0XEV2IfdbdPws8B3w6FFsJrA3pdWGdsP1Zj55VvA64JczuWQwsAV4Zt5ZkyI1LK+ntH+THm3ZnuyoiIiftZObp/xHwe2ZWRzRmf2/IvxcoD/m/B9wO4O6bgYeBt4EngdvcfdI/1ez8yhKWzJ3BYxriEZFpIHH8Ioe5+/PA8yG9nRFm37h7N/CZUfb/GvC1E61kNpkZNy6t4utPvsvO/R0smlOc7SqJiIyZ7shNw69cXIkZPPb6pL4EISJyXAr6aZhfUsCVZ8zhsdcaGBzUqxRFZOpS0E/TTUuraGjp4pWdzdmuiojImCnop+mac+dTnMzVBV0RmdIU9NNUmMzl+vMX8MSbe+jqnfSTjkRERqSgfwJuuqSK9p5+nn57T7arIiIyJgr6J+DSRWVUzi7kET15U0SmKAX9E5CTY9y0tJIX6vazp7U729URETlhCvon6MalVQw6/ItepSgiU5CC/glaNKeYS04t5dENDUSPFBIRmToU9MfgpqVVbN3XzluNepWiiEwtCvpj8IkLFpBM5Og5+yIy5Sjoj0FJYR4fP2ceazc20tuvVymKyNShoD9Gn15aRUtnH89v2ZftqoiIpE1Bf4z+y5I5VMzM529/sk0vTheRKUNBf4wSuTl85fqzeW3XQe55flu2qyMikpZ0XoxeYGavmNkbZrbZzP4s5C82s5fNrM7MHjKzZMjPD+t1YfuilGPdEfK3mNk1GWvVBLnh4ko+deEpfPuZrWysP5jt6oiIHFc6Z/o9wFXufiFwEXCtmS0Dvg58y93PAFqAW0P5W4GWkP+tUA4zO4foHbvnAtcC3zOz3HFsS1bcdcN5zJ9VwJcffJ2Onv5sV0dE5JjSeTG6u3t7WM0LHweuAh4J+WuAG0J6RVgnbL/azCzkP+juPe6+A6hjhNctTjUlhXl88+YLeb+5k7sefzvb1REROaa0xvTNLNfMNgL7gPXANuCguw+d2jYAlSFdCdQDhO2tRC9OH84fYZ/U71plZrVmVtvU1HTCDcqGZaeV85u/cDoPvlrPk2/pCZwiMnmlFfTdfcDdLwKqiM7Oz8pUhdx9tbvXuHtNRUVFpr5m3P3uxz7CeZWzuOOxTew9pIexicjkdEKzd9z9IPAc8FFgtpklwqYqYOgJZI1ANUDYXgIcSM0fYZ8pL5nI4dv/9WK6+gb4gx++oXfpisiklM7snQozmx3ShcDHgXeIgv+nQ7GVwNqQXhfWCduf9ejJZOuAW8LsnsXAEuCVcWrHpHDG3Bn86SfP4Wdb9/ODF3dmuzoiIh+SOH4RFgBrwkybHOBhd3/czN4GHjSzPwdeB+4N5e8F/sHM6oBmohk7uPtmM3sYeBvoB25z92n33sH/dulCnnu3ia8/+S5XnFHOWfNnZbtKIiLDbDI/HrimpsZra2uzXY0TdqC9h2u+/TPKi5Os/dIVFORN+ZmpIjKFmNkGd68ZaZvuyM2A8hn5/J/PXMCWvW1848kt2a6OiMgwBf0M+aUz57Lyo6dy3ws7+Ol7U2PqqYhMfwr6GXTH9WezZO4Mfv+Hb9Dc0Zvt6oiIKOhnUkFeLt++5SIOdvZy+6Ob9HpFEck6Bf0MO/eUEv7wmrN4+u29PPRq/fF3EBHJIAX9CXDrlYu54oxy/uxf32bH/o5sV0dEYkxBfwLk5Bh//ZkLSSZy+PKDr9Onl66ISJYo6E+QBSWF/OWN5/NGQytf/McNdPbqMcwiMvEU9CfQ9ecv4K4V5/Lsu/v41dUv0dTWk+0qiUjMKOhPsM99dBGrP1fDe3vbufGeF9jW1H78nURExomCfhZ87Jx5PLhqGV29A9x0z4u8urM521USkZhQ0M+SC6tn89gXr6CsKMlnv/8yj2/6INtVEpEYUNDPooXlRTz6xcu5oLKEL/3z66z+6TbdwCUiGaWgn2WlxUn+8b9fxifOX8BfPPEu/3vdZgb0AhYRyZB0nqcvGVaQl8v//dWLqSwtZPVPt/NBazd333IxhUk9kllExpfO9CeJnBzjj68/mz/71Ln8+zt7ueXvX2J/u6Z0isj4Sud1idVm9pyZvW1mm83sd0J+mZmtN7OtYVka8s3M7jazOjPbZGZLU461MpTfamYrR/vOOFt5+SL+7tcuYcueQ9z4vRfZrimdIjKO0jnT7wd+393PAZYBt5nZOcDtwDPuvgR4JqwDXEf0/tslwCrgHog6CeBO4DLgUuDOoY5CjrT83Pk88D+W0dHTz033vMiG9zWlU0TGx3GDvrvvdvfXQrqN6KXolcAKYE0otga4IaRXAPd75CVgtpktAK4B1rt7s7u3AOuBa8ezMdPJxQtLeey3Lmd2UZJbVr/EX/7bO3T06NENInJyTmhM38wWARcDLwPz3H132LQHmBfSlUDqM4QbQt5o+Ud/xyozqzWz2qameL9x6tTyYh774uX8ysWV/N1PtnP1N3/Cv77xgaZ1isiYpR30zWwG8CjwZXc/lLrNoyg0LpHI3Ve7e42711RUVIzHIae00uIk3/j0hTz6xcspn5Hktx94nc9+/2W27m3LdtVEZApKK+ibWR5RwP8nd38sZO8NwzaE5b6Q3whUp+xeFfJGy5c0XHJqKeu+dCV33XAemz84xHXf+Rl/8cQ7tGvIR0ROQDqzdwy4F3jH3f8mZdM6YGgGzkpgbUr+58MsnmVAaxgGegpYbmal4QLu8pAnacrNMT637FSe/f1f4KalVaz+6Xau/ubzrN3YqCEfEUmLHS9YmNmVwM+AN4Ght3/8MdG4/sPAQuB94GZ3bw6dxHeJLtJ2Al9w99pwrN8I+wJ8zd1/cKzvrqmp8dra2rG0KxZe39XC/1q7mTcbW1l2WhlfXXEeH5k3M9vVEpEsM7MN7l4z4rbJfIaooH98A4POg6/u4htPbqGjp59fv3wRv/OxJcwsyMt21UQkS44V9HVH7hSXm2N89rJTee4PfpHP1FRx7ws7uPqbP+GBV3bR3TeQ7eqJyCSjM/1pZmP9Qe5c+xZvNLRSVpzk1y5byK8tO5W5swqyXTURmSAa3okZd+c/tx/gvv/YyTPv7iWRY/zyBafwG1cu5rzKkmxXT0Qy7FhBX0/ZnIbMjMtPn8Plp89h5/4O/t+LO/lhbT2Pvd7IpYvL+I0rFvPxc+aRm2PZrqqITDCd6cfEoe4+Hn61nh+8sJPGg11UlRby65cv4uafq2aWLvqKTCsa3pFh/QOD/Ps7e7nvP3byys5mipO5fKammi9csYhTy4uzXT0RGQcK+jKiNxta+cELO/jXTR/QP+j83KIyPnH+Aq47b74u/IpMYQr6ckz7DnXzwCv1PL7pA7bua8cMdQAiU5iCvqRt6942fvzmbn68affhDuDUMj5xgToAkalCQV/GZLQO4Prz53Pd+QuYpw5AZFJS0JeTNtQBPPHmbt7bG3UAlyws5colc7jijDlcVD2bvFzd4C0yGSjoy7ga6gD+/Z29bP7gEO5QlMzl0sVlXHH6HC4/o5yz588iR/cBiGSFgr5kzMHOXv5z2wFe2LafF+sOsH1/BwBlxUk+elo5l59RzhWnz+HU8iKiB7CKSKbpjlzJmNlFSa47fwHXnb8AgN2tXbxQd4AX6/bzwrb9/PjN6I2albMLufz0ci47rZyLqmdz2pxi/SUgkgU605eMcXe27++IOoC6A/zn9gO0dvUBMLMgwUXVs7moejYXVs3mooWzmTMjP8s1FpkeNLwjk8LAoLOtqZ2Nuw7yev1BNtYf5L29bQwMRv8Gq0oLhzuCixfO5txTSijIy81yrUWmnpMa3jGz+4BPAvvc/byQVwY8BCwCdhK9NaslvDXrO8D1RG/N+nV3fy3ssxL4k3DYP3f3NSfTKJl6cnOMj8ybyUfmzeTmn4tel9zZ289bjYfYWN/CxvqDvPZ+C49vioaEEjnGWQtmcs6CWZw5fxZnzY/2rZipvwhExiqd1yX+PNAO3J8S9L8BNLv7X5nZ7UCpu/+RmV0P/DZR0L8M+I67XxY6iVqgBnBgA3CJu7cc67t1ph9P+w51szH8JfBGw0He3d3GgY7e4e3lxUnODB3AWfNnDqeL83WJSgRO8kzf3X9qZouOyl4B/GJIrwGeB/4o5N/vUU/ykpnNNrMFoex6d28OFVpP9A7dB060MTL9zZ1VwPJz57P83PnDeU1tPby3t41397Tx3p423t3bxkOv1tOV8naw6rJCzgx/SZxWMYPFc4o5bU4xpcXJbDRDZFIa66nRPHffHdJ7gHkhXQnUp5RrCHmj5X+Ima0CVgEsXLhwjNWT6aZiZj4VM/O54ow5w3mDg05DSxfv7jk03CFs2dPG81ua6B88/BdsaVEei+cUs3jODE6rKA7pYhaVF1OY1DUDiZeT/nvY3d3Mxu1qsLuvBlZDNLwzXseV6Scnx1hYXsTC8qIj/iroGxikoaWLHfvb2d7Uwfb9Hexo6uCFuv08+lrDEcc4paSAxRXFnFpeTFVpIVWlRWFZSMWMfN1bINPOWIP+XjNb4O67w/DNvpDfCFSnlKsKeY0cHg4ayn9+jN8tckx5uTnDZ/NXnXXkto6efnYe6GBH6Ah27O9g2/4OnnprzxHXDQDyEzlUHtURDKdnFzJnRr7uNZApZ6xBfx2wEvirsFybkv8lM3uQ6EJua+gYngL+wsxKQ7nlwB1jr7bI2BTnJzj3lBLOPeXD7wru7O2nsaWLhpYuGlo6wzJKb25s/VCnkJdrzJtVwIKSAuaXFDJ/Vj7zSwrDepRfMSOfhJ5JJJNIOlM2HyA6S59jZg3AnUTB/mEzuxV4H7g5FH+CaOZOHdGUzS8AuHuzmd0FvBrKfXXooq7IZFGUTLBk3kyWzJs54vbUTqG+pZPdrd3sae1md2sXbzYc5OnWbnr6B4/YJ8dg7syoE5g/q4C5s/KpmJEfLWfmUzEjyisvTqpzkAmhm7NExom7c7Czjz2HhjqDbva0dkXLQ9F6U1vP8F3JqcygrCg5fMF6+DMjn/IZScqKo44hSifJT+gCtIxOz94RmQBmRmlxktLiJGcvmDVque6+Afa399DU1sO+tmh5RLq9h2372mlq76FvYOSTshn5ieEOoLw4SXlxPmUzonRpUZLS4jxKCpOUFuVRWpRkVmEeubr+ICjoi0y4grzccEG46Jjl3J1DXf0c6OjhQEcvB9p7ae7opbmjh/3D6V4aWrrY1NBKc0fvEVNVU5nBrII8SovymF2UHF7ODp1CSWEeJYV5zCpMpKSjpf6qmF4U9EUmKTOjpCiPkqI8Tqs4fvmhTqKls5eWzl4OdvVxsLOXlo6w7OyL8jv7aGrv4b297Rzs7KWjd+CYx81P5Ax3BKkdwsyCBDPyE8wsiNKHP3kh/3Baf2VMHgr6ItNEaiexiOK09+vtH+RQdx+tXdHnUMryUHd/lN8Z8rqjaxbv7Wujrbuftu7+4QfmHUtxMpcZBQmK86OOojiZYEboNIrzcynOTzAzP9o+VGZoW1EyKj9ULj+Ro/snToKCvkjMJRM5zJmRP6ZHW7s73X2DtHVHHUR7Tz9t3X2hQ+gb7hjauvvp6Im2t/dE6frmTjp6++noGaC9u5/egcHjfyHRjKjiZIKi/NwjlsX5CQqTuRTl5VKUzKUgmUtRXoLCZA6FyQRFebkUJnOHyxQmQ7m8XArzomVBXu60/6tEQV9ExszMhgPp3NGvXaelt39wuGPo6O2nvbufzt4BOnr66egdoDN0EJ29UZnOngE6eqMy7T397GvrprNngK6+ATp7B+jqHUi7I0mVTORQkMiJ2pXSGRTk5XxoPT+RS35eDgWJKC8/kXPEtoK8nCPy8xOH9xlOJ3Im9CY/BX0RmRSSiRySieS4PiCvf2CQrr6oI+jqDZ1BSrqzt5+evqhM91C5voEor3eA7v6o7FDe/vbeKN0/QHffIN19A/T0D9Lbf+KdS6q8XKNguDOIOoKrz57LVz5xzjj9lzhMQV9Epq1Ebg4zc3OYWZCX0e8ZHHR6+gc/1Bl0h86kO3QMPf1R5zG0bWifnv7BkD+UN8j8ksKM1FVBX0TkJOXkHB7mmux037eISIwo6IuIxIiCvohIjCjoi4jEiIK+iEiMKOiLiMSIgr6ISIwo6IuIxMikfnOWmTURvY4RYA6wP4vVyZY4tlttjge1OXNOdfcRH8g9qYN+KjOrHe31X9NZHNutNseD2pwdGt4REYkRBX0RkRiZSkF/dbYrkCVxbLfaHA9qcxZMmTF9ERE5eVPpTF9ERE6Sgr6ISIxkLeib2bVmtsXM6szs9hG255vZQ2H7y2a2KGXbHSF/i5ldk+4xsy1Dbd5pZm+a2UYzq52gpqRtrG02s3Ize87M2s3su0ftc0loc52Z3W1mk+pN1hlq8/PhmBvDZ+4ENSdtJ9Huj5vZhvCbbjCzq1L2ma6/9bHanNnf2t0n/APkAtuA04Ak8AZwzlFlfgv425C+BXgopM8J5fOBxeE4uekcM5ufTLQ5bNsJzMl2+zLQ5mLgSuA3ge8etc8rwDLAgH8Drst2Wyegzc8DNdluX4bafTFwSkifBzTG4Lc+Vpsz+ltn60z/UqDO3be7ey/wILDiqDIrgDUh/QhwdejlVwAPunuPu+8A6sLx0jlmNmWizZPdmNvs7h3u/h9Ad2phM1sAzHL3lzz6P+R+4IZMNuIEjXubp4iTaffr7v5ByN8MFIYz5On8W4/Y5omodLaCfiVQn7LeEPJGLOPu/UArUH6MfdM5ZjZlos0ADjwd/kRclYF6n4yTafOxjtlwnGNmUybaPOQH4c/9P51swxyMX7tvAl5z9x7i81untnlIxn5rvRh96rvS3RvDuN96M3vX3X+a7UrJuPts+J1nAo8CnyM68502zOxc4OvA8mzXZaKM0uaM/tbZOtNvBKpT1qtC3ohlzCwBlAAHjrFvOsfMpky0GXcfWu4DfsTkGvY5mTYf65hVxzlmNmWizam/cxvwz0yu3xlOst1mVkX07/fz7r4tpfy0/a1HaXPGf+tsBf1XgSVmttjMkkQXONYdVWYdsDKkPw08G8b11gG3hDG/xcASoos96Rwzm8a9zWZWHM4GMLNiorOFtyagLek6mTaPyN13A4fMbFn4s/fzwNrxr/qYjXubzSxhZnNCOg/4JJPrd4aTaLeZzQZ+DNzu7i8MFZ7Ov/VobZ6Q33qir3inXKG+HniP6Or3V0LeV4FPhXQB8EOii5avAKel7PuVsN8WUq7mj3TMyfQZ7zYTzRp4I3w2T8M27wSagXai8dJzQn4N0f8I24DvEu4snyyf8W4z0ayeDcCm8Dt/hzB7azJ9xtpu4E+ADmBjymfudP6tR2vzRPzWegyDiEiM6I5cEZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEY+f/SA3DWVh7wVwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# EXERCISE 5.2\n",
    "\n",
    "# START YOUR CODE\n",
    "init = np.array([1.0,0.0]).reshape(2,1)\n",
    "it = []\n",
    "ls =  []\n",
    "\n",
    "for l in np.linspace(0.001, 0.025859, num=25):\n",
    "    xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(rosenbrock, gradient_rosenbrock, l, init, 1.0e6, eps=1.0e-6)\n",
    "    it = np.append(it, len(cost_hist))\n",
    "    ls = np.append(ls, l)\n",
    "\n",
    "plt.figure(4)\n",
    "plt.plot(ls, it)\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# END YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comments**:\n",
    "The graph plots the number of iterations needed (vertical) vs learning rate (horizontal).\n",
    "\n",
    "Compared to the previous graph, we can observe that $\\rho_{\\rm opt}$ and $\\rho_{\\rm max}$ are slightly higher, but the amount of iterations needed is higher.\n",
    "\n",
    "\n",
    "At $\\rho_{\\rm opt} = 0.025859$ the number of iterations needed is $504$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 594,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD6CAYAAABamQdMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAArlklEQVR4nO3deXyV1Z348c83+wbZDRAiAUVkEQHjbt1w32DqUq2t1Nqhtnax7XRq63TG2pmpM3VqtYvWSqf4a+vaquhgW2u1Lgg0KMgmENawh5CFJGT//v6454abkJCb5N7c50m+79crr/vc8zzPvd88XL4595zznCOqijHGmKElLtYBGGOMiTxL7sYYMwRZcjfGmCHIkrsxxgxBltyNMWYIsuRujDFDUFjJXUS+JiJrRWSNiDwlIikiMl5ElolImYg8IyJJ7thk97zM7S+O6m9gjDHmKNLbOHcRKQTeAaao6mEReRZYDFwF/EFVnxaRx4BVqvqoiHwRmK6qd4rIzcA/qOonjvUeeXl5WlxcHInfxxhjho0VK1YcUNX87vYlhPkaCUCqiLQAacAe4GLgk27/QuA+4FFgjtsGeB74qYiIHuOvSHFxMaWlpWGGYowxBkBEtve0r9dmGVXdBTwI7CCQ1GuAFUC1qra6w3YChW67ECh357a643O7CWq+iJSKSGlFRUX4v40xxphe9ZrcRSSbQG18PDAGSAeuGOgbq+rjqlqiqiX5+d1+qzDGGNNP4XSoXgJsVdUKVW0B/gCcC2SJSLBZZyywy23vAooA3P5MoDKiURtjjDmmcJL7DuAsEUkTEQFmA+uAN4Ab3DHzgJfc9iL3HLf/r8dqbzfGGBN54bS5LyPQMfo+sNqd8zjwLeDrIlJGoE19gTtlAZDryr8O3BOFuI0xxhxDr0MhB0NJSYnaaBljjOkbEVmhqiXd7bM7VI0xZgiy5G7MMPbe5krK9h+KdRgmCsK9ickYMwTd8sulAGx74OoYR2Iizfc19/Z2pa099v0GxhjjJb5P7hP/5VWuf3RJrMMwxhhP8X1yb2tXVpZXxzoMY3yn3b7xDmm+T+7GmP5pam2PdQgmioZMcm9saYt1CMb4SkNza+8HGd8aMsl9V/XhWIdgjK80NFuFaCgbMsl9a0V9rEMwxlcOh3zbtRFnQ4/vk3tuehIAH+6qiXEkxvhLaM293ppohhzfJ/fUpHgAVtmIGWP6JLTNvb7JkvtQ4/vkHpz37P0dVbS2We+/MeEKHYRgyX3o8X1yB0hPiudQYysfWO3dmLB1apZpss7Vocb3yV1VOW9iHonxwuLVe2IdjjG+EZrcaxtbYhiJiQb/J3dgZEoiV0wbzfMrdtrYXWPCdDgkue+rbYphJCYafJ/cAUTgtrPHcaixlZdW7o51OMb4Ql1IO/veGrtPZKjxfXIPdqiWjMtmyuiRPPa3zTS1WvuhMb0pP9hAXkYS2WmJ7KlpjHU4JsJ6Te4iMklEVob81IrI3SKSIyKvicgm95jtjhcReUREykTkQxGZFc1fQFEEQUT41pUns72ygYVLtkXzLY0ZErYeqGdcbjqjMlPZa8l9yAlngewNqjpDVWcApwENwAsEFr5+XVUnAq9zZCHsK4GJ7mc+8GgU4g6JL9AsA3DBSfnMPvk4fvyXTba6jDG92F7ZQHFuOqMzU6zmPgT1tVlmNrBZVbcDc4CFrnwhMNdtzwGe1IClQJaIjI5EsN1RjiR3gP/4h1NITYznC795n6r65mi9rTG+1tDcyt7aRsbnpXF8ThpbD9TbFARDTF+T+83AU267QFWDYw/3AgVuuxAoDzlnpyvrRETmi0ipiJRWVFT0MYyjXq1ja1RmCj+5ZSbbDzbwqQXLbLZIY7qxbnctAJNGjWTqmJEcbmlj6wGbn2koCTu5i0gScB3wXNd9qqoEKtFhU9XHVbVEVUvy8/P7cmqX1zm67JwT83j01lms3V3LbQuWWw3emC5W7QzMxXTq2EymjskEYO1um59pKOlLzf1K4H1V3eee7ws2t7jH/a58F1AUct5YVxYl2qlZJmj25AJ+cstMVpZXc/2jS9hmtRJjOry/o4rRmSkcNzKFiQUZpCTG8cGO6liHZSKoL8n9Fo40yQAsAua57XnASyHlt7lRM2cBNSHNNxGnGtoo09m1p47hN587k8r6Zi576C0e+9tmmm31GTPMNbW28daGCj42MQ+AxPg4Ti/OYemWyhhHZiIprOQuIunApcAfQoofAC4VkU3AJe45wGJgC1AG/BL4YsSi7UbXDtWuzhifw/995TzOPymfB179iBseW8KK7VVod+05xgwDb3y0n0NNrVx5ypFxDmefkMtHew/ZkMg+qm1s4Rd/2+zJ9WjDSu6qWq+quapaE1JWqaqzVXWiql6iqgdduarqXap6gqqeoqql0Qo+SHqsuweMzU7jl7edxk9umcm2A/Vc/+gSbnjsPXZUNkQ7NGM8RVX5xVtbKMpJ5WMn5nWUXzUtkOh///7OWIXmS//+yjp+8OpH/G3TQAeFRN4QuEM1vL+YIsK1p47hvW/P5vtzprJx3yEu/p83+fJTH/D+jqooR2mMN5Rur+KDHdX848cmkBB/5L9/cV46Z4zP4bnScvtW2we1hwNTODR6cMlC/yd3jt0s01V6cgKfPruYP959Pp85p5g3P9rPx3++hLk/e5cF72zlo721UYvVmFhSVX74xw3kpidx42lFR+2/qaSIbZUNvFtmbe/hinMZ1IOtMkMguR+jQ/VYCrNS+ZdrpvDed2bzveumUnO4he+/so4rfvw2tzy+lGf+vsNG2JghZcE7W1m+7SDfuGxSxwpmoa6ZPpoxmSn85+L1dkNTmMTVLNs9+G0nIdYBDJSqdlzg/shITmDeOcV8+qxx7K45zCsf7uHJJdv41u9XIxKY0qBkXDZzZxZSmJU6oPcyJlZKtx3kgVc/4rIpBdxyxtG1doCUxHjuuWoyX3nqA55fUc4nTj9+kKP0nziXD5o8OArP98k9UuLihLHZadx5wQl87rzxlFXU8erqvTxbWs6bGyp48M8byctIZkZRJnNmFDKjKIuinLRYh21Mr9buruELv32fwuxUfnjjqcesoFw7fTQLl2zjh3/awGVTRpHtFqA33Ytzl9KL60j4PrlH48tQQnwcJ48aycmjRvK1S0+i/GADi1fvYWV5NSvLq/nL+sD9WnkZScw6PpvTxmUzoyiLyWNGMjIlMQoRGdM/b27Yz12/fZ+RqYk8cVsJmanH/nyKCN+7biof//kSvv7sShbMO524OPu22pPglanz4Bq0vk/uaN86VPujKCeNz19wAgCtbe2s2F7FB+XVfLizmlXlNfx5XeCm3aT4OCbkpzPz+CxmFmWTk57ErHGBR2MGU2tbO79bvoPvvbyOSQUj+N/bT6dgZEpY504rzOS710zmuy+t5edvlvGliydGOVr/CnZNNHhwDVrfJ3el93HukZQQH8eZE3I5c0Ju4P1V2X+oKZDwd1Sxsryal1ft4anlR+ZOG5+XzsyiLHIzkph5fDYzj88iNz2ZpATf92cbD9q07xD3vrCG5dsOcsFJ+fzs1llkJPftv/qnzhrH37dV8eCfNzIyNZHbzi6OTrA+d9hNTGg19ygIdKjG7v1FhIKRKVx1ymiucnf8NbW2sae6kW2V9awqr2FleRVvbNhPfVMbv3x7a8e5k0ePJCc9kRlFWeSkJzNtzEjGZKVaW77pl8q6Jn63bAc/faOM+Djhv6+fzvWnjSW+H80qIsKDN55KQ3Mr33t5HSNTEpk786jJXYe94Kyz1uYeJV5rEUxOiKc4L53ivHQunHRcR/nh5jbW7q7hgx3VVNY3s2ZXDXtrG/nZG5s7nZ+UEMeEvHTGZqcyefRI15GbRWpSPAUjUshMs3Z9c8T6PbX877tbeXHlbppb25l98nE8cP108kckD+h1kxLieOgTM/jcwlLufmYlu6oP88ULT7ARYyEa3M1L9dYsE3neG13as9SkeEqKcygpzjlq39YD9eysamDd7loqDjXx4c4atlU2dHTeBiXECdnpSZxUkMFxI1IYkZLAtMJM4kWYWjiS3PRk8jKS7D/gELftQD1LNlfy8qrdvLelktTEeG48bSy3n1vMiceNiNj7jEhJ5Mk7zuCfn/+QH/5pA+UHG/j+3GkkxluTIgQqbAD1VnOPPB2EDtXBMD4vnfF56XxsYue57euaWjlY18ya3TW0tStrdtdwsK6ZleXVlB88zP5DjTz53vZO5xRmpZKSGEd8nDB9bBbpSfGMyUplfF46iQlxzCzKIikhjjgRUhKPvpnFeE97u7Kr+jAvfLCLFdureGtTBaowJjOFe648mZtPLyIrLTod98kJ8fz4EzMoyk7jp2+UsbumkZ/3ox1/KOpoc2+05B5xysBuYvK6jOQEMpITOD430A5/7aljOu1vaG6lsq6Zwy1trCyvprKumbW7a1DgUGMrr6/fR1u7UtvDh2/y6JGkJMZxfE5guTUI/HEozksHYHRmCuNy06P3C5pu1Te1srK8uqOjfvWuGg7UBRadKcxK5csXT+S6U0dTnJveaY6YaBER/unySYzNTuXeF9dw42Pv8cvbTmNs9vDuHwrW3GsOt8Q4kqP5P7n3c/qBoSItKYG0nMA/40kFPX8dLz/YQM3hFirrm/loT2D+nKqGFtbtqaWtvZ03PtpPXVNrt3NkxMcFxiPlj0hmYsGIjus9efRIct0wz+TEOGYUZZHgJtvIG5HEcSPCG3o3nDW2tLGlop5d1YdZWV7F2t217KluZMO+Iwu8F2alcuaEXGYWZXH51FEx7XC/+YzjGZWZwl2/fZ+LH/wb3587dVjfyRrsSK1qsOQeHcM5u4epKCetY3msC07qeVlDVWXt7lpqG1tAYd2eWqoamlGFDXsPccAtWdjc2s7fNm7u8XVEICs1seNb1ZTRIzvd7ViYlcr4vM5JakJ+BqMzO/9BSEmMJy9jYB2DsXCosYWawy2owuaKOvbVBuZJ33Kgnt3VjTQ0tbJqZw2HGls63bo+qWAE2emJfPHCE5gVHDbrsd//wknH8epXz+feF1fzrd+v5u1NB/j3udOi1izkVS1t7RxyQyCrG5oHPBVKpPk+ufupQ9UPRIRphZkdz88JmfO7q8aWNlpdVX9/bSMb99W5Pcr6PYc46P4QNLUGmox2Vx8GApMsLV69J+zJqQqzUkmM7/yfJiH+SN9BT8ZkpTIhL3JNSk2t7awsr6alrfM8Iq1tyqqd1R3D4hTYXX2Ylrajf7/4OKEoO5W4OOHcE3PJSU9yQ2ED90D4pR37+Nw0fvWZ03n4L5t4/K0tvFN2gPvnTOPa6aM9leCi6UBdE6qB/rKtB+qpa2plhIfuUPfHJ+lYdHBvYjJHhHbGZuRnMCE/o+P5FdNGd3dKh5rDLdSH3PjR1h5IkA1d5sXeW9PI5oq6rqdzsL6ZNzbsP6o8qF3p+OMSSSmJcd0m4BPyM5g06kiz2CWTC5hUMAIEctKSmDxmJEJgyunepgDwi8T4OP7p8klcPnUU9764mq889QGLVu7ivuumDou2+OCqVSePGsHWA/VUN7T4L7mLSBbwBDCNQMXks8AG4BmgGNgG3KSqVRL4s/0wcBXQAHxGVd+PdOBB2sMC2cbbMlMTj0pykW5L3l19ONC8FEHFuek2wqiLU8Zm8tydZ/Prd7fx4J83cNXDb3PPlZO55YyiIV2LDza1nTxqJK+u2UtVQ7OnbkAMt+b+MPBHVb1BRJKANOA7wOuq+oCI3APcA3wLuBKY6H7OBB51j1Ex3DtUTc/GZKUyhtRYhzEsJCfE8/kLTuCKaaP42jMr+c4Lq1m8eg//dcN0CrOG5r/BvtomAE4eHfjG5rVO1V7HUIlIJnA+sABAVZtVtRqYAyx0hy0E5rrtOcCTbi3VpUCWiBz7O/oADeHKgTG+Mi43nd9/4Ry+e80UVpZXc8n//I1fv7t1SC7+sbe2kcR44YT8QL9OdUPkmwEHIpwBsuOBCuB/ReQDEXlCRNKBAlXd447ZCxS47UKgPOT8na6sExGZLyKlIlJaUdH/xWWH3kfGGH8TEe44bzwvf/k8Soqzue/lddz0i/fYGDK8cyjYcbCBwqxUctMDo5mC9yF4RTjJPQGYBTyqqjOBegJNMB00sKJun/Ksqj6uqiWqWpKf3/PQvDBexzpUjfGg8Xnp/Pr2M/jhDdNZv6eWa37yDo+8vomqKHR0x8K2A/WMy00nMzWRxHjhQF1TrEPqJJzkvhPYqarL3PPnCST7fcHmFvcYHLqwCwhdx2usK4uKvi6QbYwZPPFxwo0lRbz1zxdx/sQ8fvTaRq565G1ec2sg+JWqsu1APePz0omLE/Iykqk45LPkrqp7gXIRmeSKZgPrgEXAPFc2D3jJbS8CbpOAs4CakOabiLMOVWO8Ly8jmSfmnc5Ld51LamI8//hkKXc//QE1HuuEDFdFXRP1zW0Uu2lB8jKSPVdzD3e0zJeB37qRMluA2wn8YXhWRO4AtgM3uWMXExgGWUZgKOTtEY24O1Z1N8YXTi3K4k9fO58fvbaRJ97ewtubDvD9udO4ctooXw2bLNsfuPdivLu3I39EcsfQSK8IK7mr6kqgpJtds7s5VoG7BhaWMWaoSoyP41tXnMzlU0fx3RfX8MXfvs8VU0fx0CdmkJrkj3sINuwNdA5Pdjeu5WUksXZ3TSxDOoqvJ2UO/B2xZhlj/GhGURbP3Xk237x8En9at5erH3mbZVsqYx1WWDbsPUR2WmLHgij5I5I5UNdMu4eGfPo8uQceffRtzhgTIiUxnrsuOpFffeZ02lW5+ZdL+ZcXVx81f4/XrN97iEmjRnQ0JeVlJNPWrlR7aOpffyd392hDIY3xt4smHcdLXzqPeWcX85ulO/jEL95jvZua2mta2tpZv6eWU0Im2AvW4L00YsbXyT3Iau7G+F9maiL3XTeVH910KjsONnDDo0t48YNdnmrqANi47xDNre2cMjaroyw4LbWXRsz4OrkH29yNMUPHx2eN5ZUvf4zivHTufmYlX31mJU2t3lmAevXOQMfpdKu5R8+RZhljzFAyKjOFRV86j29ePomXV+3m0wuWe2bulg931TAiJYFxuUdmgAwmd6u5R4h1qBozdMXHCXdddCIP3zyDlTuqufDBN/njmr2xDovVO2s4pTCz07j8EckJJCXEWc09UtTV3f1084Mxpm/mzCjk9184h3E5aXzhtytY8M7WmMXS1NrGR3trOWVsZqdyESEvPanXycNWbK/i/R1V0Qyxg6+TuzFmeDhlbCZPzz+by6YU8P1X1nHforUxmUZ44946WtqU6YVZR+3LSkuippehkNc/uoSP/3xJx/NILyYTytfJ3fpTjRk+UpPi+fmtp/HZc8fz6yXb+MHi9YMew4e7qgGY3qXmDpCVlhh2v4CqsrOqgYt++CbP/r289xP6wf9rqGJt7sYMF/Fxwr9eO4XW9naeeGcr7Qr3Xj2Z+LjBSQKryqvJSktkbPbRq0tlpSWGLBJ/tNDEv6+2if9cvJ7m1nbOGJ8TlVh9ndw7OlRtvIwxw8q/XTuVOBF+9e5WDtQ18fDNMwal72351oOUjMvu9r2y0pJ6rLm3tyu3PrGs4/mWijqWb63i0qkFFOelRyVWfyf3jg7VGAdijBlU8XHCfddNJTc9if95bSPHjUjm3qsnRzXB76ttZFtlA7eeOa7b/VmpiVQ3tKCqtLYrX3nqA6obWvjxzTN44NWPWLv7yB23n3SJ/tSQG6EizdfJPchyuzHD05cuPpGKuiaeeGcr+SOS+fwFJ0TtvZa6Sc3OnNB9M0p2WhKt7UpdUyurymt41Q3bPPM/XwegYGQyd19yEt/+w+qOc04blx21eH2d3K1D1ZjhTUS479qpHKhr4gevfkRmaiI3n3F8VN5r2daDZCQnMGX0yG73Z6YlAlDd0MLumsMAnFSQQf6IZP7t2qmcmJ9BXJx0JPf/vmE60wqP7piNFH8nd/dozTLGDF9xccJPbpnFocblfOeF1WSkJHDN9DERf59lWyopKc4mIb77QYZZqUeS+76awMIdi750HimJneeo/9RZx/ObpTv4+MzCiMcYyudDIYPzuVt2N2Y4i48THvvUacw6PptvPvchm/YdiujrVxxqYnNFPWeOz+3xmIyUQF25vrmVfYcayUpLPCqxA3zvummsu//yHv9IRIq/k7t7tJq7MSY9OYGf3TqL9OR47vzNCuqbWiP22su3HgR6bm8HSE9yyb2plb01TYwamdLtcfFxQlpS9BtNwkruIrJNRFaLyEoRKXVlOSLymohsco/ZrlxE5BERKRORD0VkVjR/AWOMCSoYmcIjt8xk64F67vnD6ojNHPvu5gNkJCd0msO9q/TkQC29vrmNfbWNFPSQ3AdLX2ruF6nqDFUNrqV6D/C6qk4EXnfPAa4EJrqf+cCjkQq2K+tQNcZ0dc4JeXzjssBskr9Zun3Ar6eqvLWxgrNPyCXxGE0p6cmB2nhDUysH65vJzUga8HsPxECaZeYAC932QmBuSPmTGrAUyBKR0QN4n551zApp7TLGmCO+cMEJXDQpn/tfWTfg9vdtlQ3srDrM+RPzjnlcsKmlvrmNQ40tjExJHND7DlS4yV2BP4vIChGZ78oKVHWP294LFLjtQiB0soSdrqwTEZkvIqUiUlpRUdGP0ENuYurX2caYoSouTnjwxlNJSYznuy+tGdAkY29tDOSn80/KP+ZxaUmuWaaplbqmVjKSYzsYMdzkfp6qziLQ5HKXiJwfulMDDVt9unqq+riqlqhqSX7+sS9az68ReLSKuzGmq9yMZL57zRSWbjnIw3/Z2O/XeWtjBcfnpDEu99jTBCTGx5GUEEdlXRPtemT0TKyEldxVdZd73A+8AJwB7As2t7jH/e7wXUBRyOljXVnUWG43xnTnppIibjhtLD99o6xfC243t7bz3pZKzj/p2E0yQelJ8eyrDSzY4fmau4iki8iI4DZwGbAGWATMc4fNA15y24uA29yombOAmpDmm4iy/lRjTG++e/UURqYmcv/L6/o8eqZ0+0Eamtv42MTwWhfSkhLYdyhwA9MIH9TcC4B3RGQVsBz4P1X9I/AAcKmIbAIucc8BFgNbgDLgl8AXIx6103ETk7XLGGN6kJmWyDcuPYn3tlTy1PK+zZ3++vr9JMXHcd6J4dXcM5IT2O+Rmnuv766qW4BTuymvBGZ3U67AXRGJrhd2E5MxJhy3njmOP63dx3/83zoumXwcx4UxBl1V+cv6fZxzYm7HMMfepCXHs7kiMKd7uOdEi7/vUO2Yz90YY3oWFyd8f+40mtva+eGfNoR1Ttn+OrZXNnDJ5ILeD3bSkxJodSNzYl1z93Vy72BVd2NML8bnpfPZc8fz3IqdvPHR/l6PX7w6MGVvX5J76Fwyfmhz9yy1LlVjTB987dKTOHnUCO59YTWHm9t6PK6lrZ3fLd/OxybmMSoz/GkEEkKW+7Oa+0BYs4wxpg9SEuO5f840dtc0cv8ra3s87o9r9rKvtonbzy3u0+vHhWTUwZgc7Fh8ndytQ9UY01dnjM/h8+dP4Knl5ZRuO9jtMb9eso3i3DQuPOm4Pr126Mi9uBhnV38nd1sg2xjTD1+ZPZHCrFS++vRKyg82dNr3920HWbG9innnFBMX17fcEhea3GNc6/R1cg+ymrsxpi/SkxN47FOnUdfUyqcXLKO6oRmAqvpm7n56JWOzU7mxpKiXVzlaaCqy5D4A1qFqjOmvU8Zm8sS8EnZXN3LVw2/zo9c28sknllFxqImf3zqrXx2ioRX9Plb6I87fyd06VI0xA3B6cQ6/+8czKcxO5ZHXN3Ggrolf3HYa08dm9ev1Qmvrsb5z3hbINsYMayXFOTx35znUNLSQnhw/oLVNgwk91rV28HtytwWyjTERkpk28MU1gkk91u3t4PNmmQ6xv47GGNOR1C25D5CtoWqM8ZLg2HYP5HZ/J/cgD1xHY4wJaXOPfVbydXJXWyDbGOMhR9rcYxsH+D252wLZxhgPsTb3CPPAdTTGmI6k7oWcFHZyF5F4EflARF5xz8eLyDIRKRORZ0QkyZUnu+dlbn9xlGK3DlVjjKcEk3pf56SJhr7U3L8KrA95/l/AQ6p6IlAF3OHK7wCqXPlD7riosJuYjDFe4rtmGREZC1wNPOGeC3Ax8Lw7ZCEw123Pcc9x+2dLlHo87SYmY4yX+LFD9cfAPwPt7nkuUK2qre75TqDQbRcC5QBuf407PuKs5m6M8ZIjbe6xT0q9JncRuQbYr6orIvnGIjJfREpFpLSioiKSL22MMTHhpbllwqm5nwtcJyLbgKcJNMc8DGSJSHBumrHALre9CygCcPszgcquL6qqj6tqiaqW5Ofn9yt461A1xniJr+aWUdVvq+pYVS0Gbgb+qqq3Am8AN7jD5gEvue1F7jlu/19Vo5WGXZu7By6kMcb4rkO1B98Cvi4iZQTa1Be48gVAriv/OnDPwELsmc3nbozxkmDN3QO5vW9T/qrqm8CbbnsLcEY3xzQCN0Ygtt7jcY9euJDGGCN+vInJy2wopDHGCzpGy3ggJ/k6uVuHqjHGS7wwSibI38m9o0M1xoEYYwzemHYgyN/J3TpUjTEe4qWK5tBI7h66oMaY4csLQyCDfJ3cj/DOBTXGDF8eapXxd3JXrEfVGOMdVnOPEGuWMcZ4iZfulvd1cg/yzuU0xgxn1iwTIbZAtjHGS6xZJsK8czmNMcOZ1dwjxDpUjTFe4qVWBH8nd+tQNcZ4iDXLRIjNCmmM8RJrlokQWyDbGOMlVnOPNO9cT2PMMOah3O7v5G7dqcYYL7Gae4TYrJDGGC+J81BG7TUUEUkRkeUiskpE1orI91z5eBFZJiJlIvKMiCS58mT3vMztL45e+LZAtjHGO/xWc28CLlbVU4EZwBUichbwX8BDqnoiUAXc4Y6/A6hy5Q+546LCau7GGC/xUkWz1+SuAXXuaaL7UeBi4HlXvhCY67bnuOe4/bMlSr+xDYU0xniJ74ZCiki8iKwE9gOvAZuBalVtdYfsBArddiFQDuD21wC53bzmfBEpFZHSioqKAf0SxhjjBX5rlkFV21R1BjAWOAM4eaBvrKqPq2qJqpbk5+f38zUCjzbO3RjjBb6ruQepajXwBnA2kCUiCW7XWGCX294FFAG4/ZlAZSSC7SYeAu8TjVc3xpi+8k4yCme0TL6IZLntVOBSYD2BJH+DO2we8JLbXuSe4/b/VYNZOMI62tyj8eLGGNNHXqq5J/R+CKOBhSIST+CPwbOq+oqIrAOeFpF/Bz4AFrjjFwD/T0TKgIPAzVGIGzjSLGPZ3RjjBV5qc+81uavqh8DMbsq3EGh/71reCNwYkejCZG3uxhgvCN7E5IUc76H7qfrO5nM3xniJr8a5e5rN526M8RAvNcv4Orlbk7sxxku81KHq7+RuC2QbYzzEau4R5qHraYwZxryUi3yd3K1D1RjjJcGauxdyvL+Tu80KaYzxEGuWiRCbFdIY4yXWoRohareoGmM8xEuDO3yd3IM8dD2NMcOY1dwjxLpTjTFeYm3ukWIdqsYYD7HkHiFqC2QbYzzES6nI38ndau7GGA+xmnuEeeh6GmOGsTgPZVQPhdJ30VnfyRhj+sdq7hFyZJS7dy6oMWb4sqGQEWILZBtjvMRLgzvCWSC7SETeEJF1IrJWRL7qynNE5DUR2eQes125iMgjIlImIh+KyKxoBW+tMsYYL/FOag+v5t4KfENVpwBnAXeJyBTgHuB1VZ0IvO6eA1wJTHQ/84FHIx51Fx76Y2mMGcZ8VXNX1T2q+r7bPgSsBwqBOcBCd9hCYK7bngM8qQFLgSwRGR3pwAPxRONVjTHG//rU5i4ixcBMYBlQoKp73K69QIHbLgTKQ07b6cq6vtZ8ESkVkdKKioq+xu24NndPfRkyxpjYCzu5i0gG8HvgblWtDd2ngZ7NPtWjVfVxVS1R1ZL8/Py+nBryGsHY+nW6McYMWWEldxFJJJDYf6uqf3DF+4LNLe5xvyvfBRSFnD7WlUWczedujDHdC2e0jAALgPWq+qOQXYuAeW57HvBSSPltbtTMWUBNSPNNVFizjDHGdJYQxjHnAp8GVovISlf2HeAB4FkRuQPYDtzk9i0GrgLKgAbg9kgGHMo6VI0xpnu9JndVfYeeh2/O7uZ4Be4aYFxhOTIr5GC8mzHG+IfP71ANPFpuN8Z4QYKbf2BkamKMIwmvWcazrEPVGOMlRTlp/Os1U7jqlKjc2tMnvk7uR1h2N8Z4w2fPGx/rEADfN8tYj6oxxnTH18k9yJpljDGmM18nd+tQNcaY7vk7udsC2cYY0y1fJ/cgS+3GGNOZr5O79acaY0z3hkRyt1YZY4zpzN/J3T3axGHGGNOZv5O7LZBtjDHd8nVyN8YY0z1fJ3frTzXGmO75OrljHarGGNMtXyd3u4nJGGO65+/kbtMPGGNMt8JZQ/VXIrJfRNaElOWIyGsissk9ZrtyEZFHRKRMRD4UkVnRDP5IPIPxLsYY4x/h1Nx/DVzRpewe4HVVnQi87p4DXAlMdD/zgUcjE2b3rEPVGGO612tyV9W3gINdiucAC932QmBuSPmTGrAUyBKRqC1JcqRZxqruxhgTqr9t7gWqusdt7wUK3HYhUB5y3E5XdhQRmS8ipSJSWlFR0a8gbIFsY4zp3oA7VDVwm2ifW0hU9XFVLVHVkvz8/H6+d+DRcrsxxnTW3+S+L9jc4h73u/JdQFHIcWNdWXRZdjfGmE76m9wXAfPc9jzgpZDy29yombOAmpDmm4izDlVjjOleQm8HiMhTwIVAnojsBP4NeAB4VkTuALYDN7nDFwNXAWVAA3B7FGI+IjhxmFXdjTGmk16Tu6re0sOu2d0cq8BdAw0qXB1T/lpuN8aYTuwOVWOMGYJ8ndyDbG4ZY4zpzNfJXW0RVWOM6Za/k7t7tHq7McZ05u/kbvO5G2NMt/yd3N2jDYU0xpjOfJ3cO1huN8aYTnyd3K1D1Rhjuufr5B5kbe7GGNOZr5O73cRkjDHd83dytwWyjTGmW75O7kGW2o0xpjNfJ3frTzXGmO75O7m7R2uVMcaYznyd3CfkpXP1KaOJj7PsbowxoXqdz93LLps6isumjop1GMYY4zm+rrkbY4zpniV3Y4wZgqKS3EXkChHZICJlInJPNN7DGGNMzyKe3EUkHvgZcCUwBbhFRKZE+n2MMcb0LBo19zOAMlXdoqrNwNPAnCi8jzHGmB5EI7kXAuUhz3e6sk5EZL6IlIpIaUVFRRTCMMaY4StmHaqq+riqlqhqSX5+fqzCMMaYISkayX0XUBTyfKwrM8YYM0gk0gteiEgCsBGYTSCp/x34pKquPcY5FcD2fr5lHnCgn+cOJoszsvwQpx9iBIszkgY7xnGq2m3TR8TvUFXVVhH5EvAnIB741bESuzun3+0yIlKqqiX9PX+wWJyR5Yc4/RAjWJyR5KUYozL9gKouBhZH47WNMcb0zu5QNcaYIWgoJPfHYx1AmCzOyPJDnH6IESzOSPJMjBHvUDXGGBN7Q6HmbowxpgtL7sYYMwR5Orn3NrukiCSLyDNu/zIRKQ7Z921XvkFELo9xnF8XkXUi8qGIvC4i40L2tYnISvezKIYxfkZEKkJi+VzIvnkissn9zItWjGHG+VBIjBtFpDpk32Bdy1+JyH4RWdPDfhGRR9zv8KGIzArZN5jXsrc4b3XxrRaRJSJyasi+ba58pYiUxjjOC0WkJuTf9l9D9g3KDLRhxPjNkPjWuM9ijts3aNeyE1X15A+BMfKbgQlAErAKmNLlmC8Cj7ntm4Fn3PYUd3wyMN69TnwM47wISHPbXwjG6Z7XeeRafgb4aTfn5gBb3GO2286OVZxdjv8ygfsoBu1auvc5H5gFrOlh/1XAq4AAZwHLBvtahhnnOcH3JzCL67KQfduAPI9czwuBVwb6eYlmjF2OvRb4ayyuZeiPl2vu4cwuOQdY6LafB2aLiLjyp1W1SVW3AmXu9WISp6q+oaoN7ulSAlMyDKaBzNR5OfCaqh5U1SrgNeAKj8R5C/BUlGLpkaq+BRw8xiFzgCc1YCmQJSKjGdxr2WucqrrExQGx+VwG4+jtevZk0Gag7WOMMflcduXl5B7O7JIdx6hqK1AD5IZ57mDGGeoOArW6oBQ3O+ZSEZkbhfgg/Bivd1/TnxeR4PxAnryWrmlrPPDXkOLBuJbh6On3GMxr2VddP5cK/FlEVojI/BjFFOpsEVklIq+KyFRX5rnrKSJpBP5g/z6kOCbX0tcLZPuNiHwKKAEuCCkep6q7RGQC8FcRWa2qm2MQ3svAU6raJCKfJ/CN6OIYxBGum4HnVbUtpMwr19JXROQiAsn9vJDi89y1PA54TUQ+crXXWHifwL9tnYhcBbwITIxRLL25FnhXVUNr+TG5ll6uuYczu2THMRKYsCwTqAzz3MGMExG5BLgXuE5Vm4LlqrrLPW4B3gRmxiJGVa0MiesJ4LRwzx3MOEPcTJevvoN0LcPR0+/huRlTRWQ6gX/vOapaGSwPuZb7gReIXrNmr1S1VlXr3PZiIFFE8vDg9eTYn8vBvZaD3cgf7g+BbxVbCHz1DnaWTO1yzF107lB91m1PpXOH6hai16EaTpwzCXT8TOxSng0ku+08YBNR6BAKM8bRIdv/ACx12znAVhdrttvOidW1dMedTKCTSgb7Woa8XzE9dwBeTecO1eWDfS3DjPN4Av1R53QpTwdGhGwvAa6IYZyjgv/WBBLjDndtw/q8DEaMbn8mgXb59Fhey473How3GcDFvIrA9MGbgXtd2f0Ear8AKcBz7gO6HJgQcu697rwNwJUxjvMvwD5gpftZ5MrPAVa7D+Vq4I4YxvgDYK2L5Q3g5JBzP+uucRlweyyvpXt+H/BAl/MG81o+BewBWgi0894B3Anc6fYLgXWEN7tYSmJ0LXuL8wmgKuRzWerKJ7jruMp9Ju6NcZxfCvlsLiXkj1F3n5dYxOiO+QyBgRyh5w3qtQz9sekHjDFmCPJym7sxxph+suRujDFDkCV3Y4wZgiy5G2PMEGTJ3RhjhiBL7sYYMwRZcjfGmCHo/wMoNhJAT0GgJwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# EXERCISE 5.3\n",
    "\n",
    "# START YOUR CODE\n",
    "\n",
    "init = np.array([0.0,0.0]).reshape(2,1)\n",
    "it = []\n",
    "init_vals =  []\n",
    "for i in np.linspace(0, 1.823, num=500):\n",
    "    init = np.array([i,0.0]).reshape(2,1)\n",
    "    xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(rosenbrock, gradient_rosenbrock, 0.02, init, 1.0e6, eps=1.0e-6)\n",
    "    it = np.append(it, len(cost_hist))\n",
    "    init_vals = np.append(init_vals, i)\n",
    "\n",
    "plt.figure(5)\n",
    "plt.plot(init_vals, it)\n",
    "plt.show()\n",
    "\n",
    "# END YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comments:**\n",
    "The graph shows the number of iterations (verical) vs the initial values (horizontal). It can be observed that the number of iterations needed drops significantly at $(0,0)$ and $(1.75,0)$ and rises significantly at $(0,0)$ and $(1.25,0)$. Those points could be saddlepoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 605,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWj0lEQVR4nO3dfZRcdX3H8c93n3eT7CYhC0lDIOFBKVEJNnKC0IqhKmIVsA8HWzmo2IiVVo+WVqVHsccW26octR49UVDawxF58IFSOBgllqoluNGQhCd5hmAISwgzeZjdndn99o+5s9xsdrKzyd6Zub/7fp0zJzP3YeY7N3M++eV3f/d3zd0FAEiflkYXAAA4NAQ4AKQUAQ4AKUWAA0BKEeAAkFJt9fywBQsW+NKlS+v5kQCQehs3bnzB3fsnLq9rgC9dulQDAwP1/EgASD0ze2qy5XShAEBKEeAAkFIEOACkFAEOAClFgANASk0Z4GbWZWb3mtl9Zna/mX0mWv5tM3vCzDZFjxWJVwsAGFfLMMJhSavdfY+ZtUv6mZndEa273N1vTq48AEA1Uwa4l+eb3RO9bI8ezEELAFO48ZfPaNuufZKkC157tJYtmDWj719TH7iZtZrZJknPS1rn7huiVf9kZpvN7Goz66yy7xozGzCzgcHBwZmpGgCaXK5Q1N/dsllfvutRfWX9o3pq594Z/4yaAtzdR919haSjJZ1mZq+S9AlJJ0l6naT5kv6+yr5r3X2lu6/s7z/gSlAACFJuX1GS9Pk/PUVPXPU2nfXKI2f8M6Y1CsXdX5K0XtI57r7dy4YlfUvSaTNeHQCkVK5QDvDeruRmLKllFEq/mc2NnndLepOkh8xsUbTMJJ0vaWtiVQJAyuSHygHe192e2GfU8k/DIknXmVmryoF/o7vfZmZ3mVm/JJO0SdKliVUJAClTaYH39TQwwN19s6RTJ1m+OpGKACAA+fEulOQCnCsxASAB4y3wBLtQCHAASECuUFRri6mnozWxz6jrDR0AJG/nnmGNOtfaNdqO/LD6uttVHueRDAIcCMhNA8/o8ps3N7oMRE44cnai70+AAwF5/IW9amsxXfmO5Y0uBZJevbgv0fcnwIGA5ApF9XW3692rjm10KagDTmICAakEOLKBAAcCki8U1UuAZwYBDgSEAM8WAhwISH6oRBdKhhDgQEDKfeCMTcgKAhwIhLuXu1ASnHsDzYV/qhGE7bmCrrr9IQ2XRhtdSsOMuVQac7pQMoQARxD+95EXdOt9v9Xx/bPU3prd/1i+anGvVh13RKPLQJ0Q4AhCZerO733wjETnXwaaSXabKghKvlCUmTQnwdtXAc2GAEcQ8kMlzelsU0tLcjO/Ac2GAEcQclzAggwiwBGEPHOAIIMIcASBSZyQRQQ4gpAf4gIWZA+n7FFVrlDUlm25RpdRk517RnTqEgIc2UKAo6rP3vaAbtq4rdFl1GxhX1ejSwDqigBHVTt2D+vEI2frn9/56kaXMqUWk5b/TrK3rwKaDQGOqnKFohb2del1S+c3uhQAk+AkJqrazcgOoKkR4KiKoXlAcyPAMSl35+pGoMkR4JhUoTjK3NJAkyPAMalcND0rF8cAzYsAx6TyhZIk0QIHmhjDCFPkG3c/rus3PFWXzxoujUkiwIFmRoCnyLoHd2jP8KjOOKE+t8ya1dmmFcfMrctnAZg+AjxF8oWiViyZqy9deGqjSwHQBOgDTxHmvAYQR4CnSH6oRIADGEeAp0RpdEx7hkvq7abXC0AZAZ4Su4cY1gdgf1MGuJl1mdm9Znafmd1vZp+Jli8zsw1m9qiZfdfMOpIvN7sqF9YQ4AAqammBD0ta7e6nSFoh6RwzWyXpXyRd7e4nSNol6ZLEqoTyQ1wZCWB/U3aourtL2hO9bI8eLmm1pD+Pll8n6UpJX5v5EtPt6Z379KMHnjvs93ly515JUl8PAQ6grKYzYmbWKmmjpBMkfVXSY5JecvdStMk2SYur7LtG0hpJOuaYYw633tT5yl2PzNhtyTrbWrRkXs+MvBeA9KspwN19VNIKM5sr6fuSTqr1A9x9raS1krRy5Uo/hBpTbde+EZ20cI5uvPT0w36vjtYWdbW3zkBVAEIwrTFp7v6Sma2XdLqkuWbWFrXCj5b0bBIFpl2uUNS8ng76rgHMuFpGofRHLW+ZWbekN0l6UNJ6SX8SbXaxpB8mVGOqcVcbAEmppQW+SNJ1UT94i6Qb3f02M3tA0g1m9llJv5Z0TYJ1pla+wMU3AJJRyyiUzZIOmD3J3R+XdFoSRYWEFjiApHAlZoJGSmMqFEfp/waQCAI8QZWLbxi7DSAJdM5OU3F0TGNe22jInXtGJHH1JIBkEODT8OMHdmjNfw5obJqj2efSAgeQAAJ8Gh56Lq8xl/72za+QmdW0T09Hq04/vj63QAOQLQT4NOSHSupqb9Flq09sdCkAwEnM6cjtY0gggOZBgE9DrlDkhCSApkGAT0N+iBY4gOZBgE9DrlBULwEOoEkQ4NNACxxAMyHApyG3r6jeLgbuAGgOpFHM87uH9KUfP6KR0tik63cPl2iBA2gaBHjMTx8e1PUbntZRvZ1qneRCnaPndet1y+Y3oDIAOBABHpMvlCefWvfRNzBcEEDTow88JlcoqsWk2R38uwag+RHgMblCUXO62tXSUts8JwDQSAR4TJ675wBIEQI8htufAUgTAjymfKUl/d8A0oEAj8kPMc4bQHpktrm5a++Intm1b79lL+4dYfgggNTIbIBf/K17tXlb7oDlR87pbEA1ADB9mQ3w375U0Fmv7NdFq44dX9ZixpWWAFIjkwHu7soVijppYa/O/t2jGl0OABySTJ7EHCqOqTjqnLAEkGqZDPBcNOcJQwYBpFkmAzw/VA5wWuAA0iyTAV5pgRPgANIskwFemTaWMd8A0iyTAU4LHEAIUnEW7+v/85ju2LJdktTR1qKr3vkanXDk7Gm9x/89tlP/eudDGhtzvbBnRBIBDiDdUtECn9XRqnmzOjSrs02/fHKXfvnki9N+j/UPP6/N23KaN6tDJx41WxetOlZzewhwAOmVihb4Racv1UWnL9Xe4ZKWf/rO8T7s6cgXijpiVoe+/d7TEqgQAOovFS3wip6OVrW12Hgf9nQw1zeA0KQqwM1Mvd3thxzgvQQ4gICkKsCl8onH/FBp2vvlh2iBAwhL6gK8t6vt0FvgXano8geAmkwZ4Ga2xMzWm9kDZna/mX04Wn6lmT1rZpuix7nJlyv1drcf4klM7rYDICy1NElLkj7m7r8yszmSNprZumjd1e7++eTKO1Bvd7ue3VWY1j5jY678EH3gAMIyZYC7+3ZJ26Pnu83sQUmLky6smr7udg3uHtZNA8/olCVz9Yqj5hx0+41P7dKD2/Ny58IdAGGZVh+4mS2VdKqkDdGiy8xss5lda2bzquyzxswGzGxgcHDw8KqVtPSIHu0eLunymzfr8pvum3L7937rXv3DD7ZKko6Z33PYnw8AzaLmADez2ZJukfQRd89L+pqk4yWtULmF/oXJ9nP3te6+0t1X9vf3H3bBf/n7x+nnH1+tc5Yv1M69IwfddqQ0pvxQSe8/c5k2fPJsvXn5wsP+fABoFjUFuJm1qxze17v79yTJ3Xe4+6i7j0n6hqS6XOJoZlo8t1sL+7qmHI1Smfd7yfweHdXbVY/yAKBuahmFYpKukfSgu38xtnxRbLMLJG2d+fKq6+1u157hksbGvOo2eWYdBBCwWkahnCHpIklbzGxTtOyTkt5lZiskuaQnJX0ggfqq6u1qk7u0e6ikviqTUnHrNAAhq2UUys8k2SSrbp/5cmpXaVXnh4pTBjgtcAAhSt2VmBWVUD5YP3jlknsCHECIUhvglYtyDnZVZo5bpwEIWGoDvKYW+HgfOAEOIDypDfDK3XT+5oZfa+3djx2wftfeEf3bnQ/LTOpqb613eQCQuNQG+MLeLn3qj05Wb1e7fv30Swesf+rFfZKk81c07Kp/AEhUagPczPS+M5dp2YJZk3ajVJa9e9Ux9S4NAOoitQFe0VflDj2cwAQQutQHeG93+/gl83FchQkgdKkP8L7uduX2HaQFToADCFTqA7y3q027J5kTJT9UVEdbCyNQAAQr/QHe3V6eE2V4/xsd5wvcxBhA2FIf4H1VrsjMF0rcxBhA0FIf4L2TXJH5wp5h/feW7bTAAQQt9QE+WQv8zvufkyS9cmFvQ2oCgHpIfYBXxnnHhxK+FI1K+fTbT25ITQBQD6kP8Mpc4PEulHyBESgAwpf6AK+cqNwvwIeKXIEJIHipD/DZnW1qbTHlCy8PI8wViurjNmoAApf6ADcz9Xa1TehCKTECBUDwUh/g0oHzoeQKRS6hBxC8IAJ84oyE+SGuwgQQviACvLerfb9x4LkCJzEBhC+IAI+3wMfGnHlQAGRCEAHe292uXDQKZe9ISWPOPOAAwhdIgLeNn8R8eR5whhECCFsQAd7X3a6R0piGiqPj48FpgQMIXRABPj4fSqHInXgAZEYQAd4Xm1K20pXCKBQAoQsiwONzgue4mTGAjAgiwMfnBB8qvnw3+h4CHEDYggrwXKEc4GbS7A5GoQAIWxABXplSNl8ojV+F2dJiDa4KAJIVRoDHWuB7R0Y1q4MbOQAIXxAB3t7aop6OVuUKRQ2XxrgTD4BMCCLApXI/eL5Q1HBxVB1twXwtAKgqmKTr7SpPaDVECxxARgQzVKMvuqmDu9RJCxxABgSTdJUZCYdLY+qkBQ4gA6YMcDNbYmbrzewBM7vfzD4cLZ9vZuvM7JHoz3nJl1tdb3eb8oWihoqjtMABZEItSVeS9DF3P1nSKkkfMrOTJX1c0k/c/URJP4leN0zlJOYIfeAAMmLKAHf37e7+q+j5bkkPSlos6TxJ10WbXSfp/IRqrElvV7t2D5e0b4QWOIBsmFbSmdlSSadK2iDpKHffHq16TtJRVfZZY2YDZjYwODh4OLUe1JzoaswX944Q4AAyoeakM7PZkm6R9BF3z8fXubtL8sn2c/e17r7S3Vf29/cfVrEH0xZdOj8yOqbONrpQAISvpgA3s3aVw/t6d/9etHiHmS2K1i+S9HwyJdamNTb3SVc7LXAA4atlFIpJukbSg+7+xdiqWyVdHD2/WNIPZ7682sUnr6IFDiALarmQ5wxJF0naYmabomWflPQ5STea2SWSnpL0Z4lUWKMWiwU4LXAAGTBlgLv7zyRVm5v17Jkt59C1xgOck5gAMiCYpGvZrw+cLhQA4QsmwFtj34QWOIAsCCbp9usD5yQmgAwINMCD+VoAUFUwSRcfB97Wyv0wAYQvmACPt8DbWoL5WgBQVTBJF78JPfkNIAuCibp4F0p8TDgAhCqYAG+hDxxAxgQT4PFWdwstcAAZEEyAx0M73p0CAKEKJ8Bj34QAB5AFwQR4Ky1wABkTToAzCgVAxgQT4EYLHEDGBBPg+7XACXAAGRBOgDOMEEDGBBPg8VEoXMgDIAuCCXBOYgLImmACPN5t0kIfOIAMCDLA2whwABkQTIDHu1BogQPIgnAC3OgDB5AtwQR4PLMZBw4gC4IJcC7kAZA1YQY4XSgAMiCYAGcYIYCsCSjAG10BANRXMAFOvzeArAkmwOk2AZA1wQQ4Jy4BZE0wAc4UsgCyJpwAD+abAEBtgok9ulAAZE04Ac5JTAAZE0yAGy1wABkTTIADQNYQ4ACQUlMGuJlda2bPm9nW2LIrzexZM9sUPc5NtkwAwES1tMC/LemcSZZf7e4rosftM1sWAGAqUwa4u98t6cU61AIAmIbD6QO/zMw2R10s82asIgBATQ41wL8m6XhJKyRtl/SFahua2RozGzCzgcHBwUP8OADARIcU4O6+w91H3X1M0jcknXaQbde6+0p3X9nf33+odQIAJjikADezRbGXF0jaWm1bAEAy2qbawMy+I+ksSQvMbJukT0s6y8xWSHJJT0r6QHIlAgAmM2WAu/u7Jll8TQK1AACmgSsxASClCHAASCkCHABSigAHgJSa8iRmmnzpwhU6YlZno8sAgLoIKsDPW7G40SUAQN3QhQIAKUWAA0BKEeAAkFIEOACkFAEOAClFgANAShHgAJBSBDgApJS5e/0+zGxQ0lOHuPsCSS/MYDlJoc6Zk4YaJeqcaWmos941HuvuB9zSrK4BfjjMbMDdVza6jqlQ58xJQ40Sdc60NNTZLDXShQIAKUWAA0BKpSnA1za6gBpR58xJQ40Sdc60NNTZFDWmpg8cALC/NLXAAQAxBDgApFRTBLiZnWNmD5vZo2b28UnWd5rZd6P1G8xsaWzdJ6LlD5vZWxpY40fN7AEz22xmPzGzY2PrRs1sU/S4Nakaa6zzPWY2GKvn/bF1F5vZI9Hj4gbXeXWsxt+Y2UuxdXU5nmZ2rZk9b2Zbq6w3M/ty9B02m9lrY+vqeSynqvMvovq2mNkvzOyU2Lono+WbzGygwXWeZWa52N/tp2LrDvp7qWONl8fq2xr9FudH6+p2LMe5e0MfklolPSbpOEkdku6TdPKEbf5K0tej5xdK+m70/ORo+05Jy6L3aW1QjW+U1BM9/2Clxuj1niY6lu+R9O+T7Dtf0uPRn/Oi5/MaVeeE7f9a0rUNOJ5/IOm1krZWWX+upDskmaRVkjbU+1jWWOfrK58v6a2VOqPXT0pa0CTH8yxJtx3u7yXJGids+3ZJdzXiWFYezdACP03So+7+uLuPSLpB0nkTtjlP0nXR85slnW1mFi2/wd2H3f0JSY9G71f3Gt19vbvvi17eI+noBOqYSi3Hspq3SFrn7i+6+y5J6ySd0yR1vkvSdxKqpSp3v1vSiwfZ5DxJ/+Fl90iaa2aLVN9jOWWd7v6LqA6pcb/NWo5nNYfzu56WadbYkN9lXDME+GJJz8Reb4uWTbqNu5ck5SQdUeO+9aox7hKVW2YVXWY2YGb3mNn5CdRXUWudfxz9l/pmM1syzX1nQs2fFXVFLZN0V2xxvY7nVKp9j3oey+ma+Nt0ST8ys41mtqZBNcWdbmb3mdkdZrY8WtZ0x9PMelT+R/mW2OK6H8ugbmrcDMzs3ZJWSnpDbPGx7v6smR0n6S4z2+LujzWmQv2XpO+4+7CZfUDl/9msblAttbhQ0s3uPhpb1kzHMzXM7I0qB/iZscVnRsfySEnrzOyhqBXaCL9S+e92j5mdK+kHkk5sUC1Tebukn7t7vLVe92PZDC3wZyUtib0+Olo26TZm1iapT9LOGvetV40ysz+UdIWkd7j7cGW5uz8b/fm4pJ9KOjWBGmuq0913xmr7pqTfq3XfetYZc6Em/De1jsdzKtW+Rz2PZU3M7DUq/32f5+47K8tjx/J5Sd9XMl2QNXH3vLvviZ7fLqndzBaoCY+nDv67rN+xrGeHe5UTAW0qn+RZppdPUCyfsM2HtP9JzBuj58u1/0nMx5XMScxaajxV5RMtJ05YPk9SZ/R8gaRHlNwJmFrqXBR7foGke6Ln8yU9EdU7L3o+v1F1RtudpPKJIWvE8Yw+Y6mqn3R7m/Y/iXlvvY9ljXUeo/L5oddPWD5L0pzY819IOqeBdS6s/F2rHH5PR8e2pt9LPWqM1vep3E8+q5HH0t0bH+DRFz5X0m+iALwiWvaPKrdkJalL0k3Rj/BeScfF9r0i2u9hSW9tYI0/lrRD0qbocWu0/PWStkQ/ui2SLmnwsbxK0v1RPeslnRTb933RMX5U0nsbWWf0+kpJn5uwX92Op8otrO2Siir3u14i6VJJl0brTdJXo++wRdLKBh3Lqer8pqRdsd/mQLT8uOg43hf9Jq5ocJ2XxX6b9yj2D85kv5dG1Bht8x6VB0/E96vrsaw8uJQeAFKqGfrAAQCHgAAHgJQiwAEgpQhwAEgpAhwAUooAB4CUIsABIKX+H0FNRfAI3i8AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# EXERCISE 5.4\n",
    "init = np.array([0.0,0.0]).reshape(2,1)\n",
    "it = []\n",
    "init_vals =  []\n",
    "for i in np.linspace(0, 1.823, num=500):\n",
    "    init = np.array([0.0, i]).reshape(2,1)\n",
    "    xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(rosenbrock, gradient_rosenbrock, 0.02, init, 1.0e6, eps=1.0e-6)\n",
    "    it = np.append(it, len(cost_hist))\n",
    "    init_vals = np.append(init_vals, i)\n",
    "\n",
    "plt.figure(5)\n",
    "plt.plot(init_vals, it)\n",
    "plt.show()\n",
    "\n",
    "#init = np.array([0.0,0.0]).reshape(2,1)\n",
    "#it = []\n",
    "#init_vals =  []\n",
    "#jnit_vals =  []\n",
    "#for i in np.linspace(0, 1.823, num=50):\n",
    "#    for j in np.linspace(0, 1.823, num=50):\n",
    "#        init = np.array([i,j]).reshape(2,1)\n",
    "#        xopt, cost_hist, learning_speed_hist = vanilla_gradient_descent(rosenbrock, gradient_rosenbrock, 0.02, init, 1.0e6, eps=1.0e-6)\n",
    "#        it = np.append(it, len(cost_hist))\n",
    "#        init_vals = np.append(init_vals, i)\n",
    "#       jnit_vals = np.append(jnit_vals, j)\n",
    "\n",
    "#plt.figure(6)\n",
    "#ax = plt.axes(projection='3d')\n",
    "#ax.plot3D(init_vals, jnit_vals, it)\n",
    "#plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comment:**\n",
    "\n",
    "Compared to the x-values, the y-values have only one point, at around $(0.0,0.1)$, where there could be a sattlepoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}