2- approx.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Function approximation by global functions\n",
    "<div id=\"ch:approx:global\"></div>\n",
    "\n",
    "Many successful numerical solution methods for differential equations,\n",
    "including the finite element method,\n",
    "aim at approximating the unknown function by a sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:u\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    " u(x) \\approx \\sum_{i=0}^N c_i{\\psi}_i(x),\n",
    "\\label{fem:u} \\tag{1}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where ${\\psi}_i(x)$ are prescribed functions and $c_0,\\ldots,c_N$\n",
    "are unknown coefficients to be determined.\n",
    "Solution methods for differential equations\n",
    "utilizing ([1](#fem:u)) must\n",
    "have a *principle* for constructing $N+1$ equations to\n",
    "determine $c_0,\\ldots,c_N$. Then there is a *machinery* regarding\n",
    "the actual construction of the equations for $c_0,\\ldots,c_N$, in a\n",
    "particular problem. Finally, there is a *solve* phase for computing\n",
    "the solution $c_0,\\ldots,c_N$ of the $N+1$ equations.\n",
    "\n",
    "Especially in the finite element method, the machinery for\n",
    "constructing the discrete equations to be implemented on a computer is\n",
    "quite comprehensive, with many mathematical and implementational\n",
    "details entering the scene at the same time. From an ease-of-learning\n",
    "perspective it can therefore be wise to introduce the computational machinery\n",
    "for a trivial equation: $u=f$. Solving this equation with $f$ given\n",
    "and $u$ of the form ([1](#fem:u)), means that we seek an approximation\n",
    "$u$ to $f$.  This approximation problem has the advantage of\n",
    "introducing most of the finite element toolbox, but without involving\n",
    "demanding topics related to differential equations (e.g., integration\n",
    "by parts, boundary conditions, and coordinate mappings).  This is the\n",
    "reason why we shall first become familiar with finite element\n",
    "*approximation* before addressing finite element methods for\n",
    "differential equations.\n",
    "\n",
    "First, we refresh some linear algebra concepts about approximating\n",
    "vectors in vector spaces. Second, we extend these concepts to\n",
    "approximating functions in function spaces, using the same principles\n",
    "and the same notation.  We present examples on approximating functions\n",
    "by global basis functions with support throughout the entire\n",
    "domain. That is, the functions are in general nonzero on the entire\n",
    "domain.  Third, we introduce the finite element type of basis\n",
    "functions globally.  These basis functions will later,\n",
    "in [Function approximation by finite elements](#ch:approx:fe),\n",
    "be used with local support\n",
    "(meaning that each function is nonzero except in a small part of the domain)\n",
    "to enhance stability and efficiency.\n",
    "We explain all the details of the\n",
    "computational algorithms involving such functions.  Four types of\n",
    "approximation principles are covered: 1) the least squares method, 2)\n",
    "the $L_2$ projection or Galerkin method, 3) interpolation or\n",
    "collocation, and 4) the regression method.\n",
    "\n",
    "# Approximation of vectors\n",
    "<div id=\"fem:approx:vec\"></div>\n",
    "\n",
    "We shall start by introducing two fundamental methods for\n",
    "determining the coefficients $c_i$ in ([1](#fem:u)). These methods\n",
    "will be introduced for\n",
    "approximation of vectors. Using vectors in vector\n",
    "spaces to bring across the ideas is believed to be more intuitive\n",
    "to the reader than starting directly with functions in function spaces.\n",
    "The extension from vectors to functions will be trivial as soon as\n",
    "the fundamental ideas are understood.\n",
    "\n",
    "\n",
    "The first method of approximation is called the *least squares method*\n",
    "and consists in finding $c_i$ such that the difference $f-u$, measured\n",
    "in a certain norm, is minimized. That is, we aim at finding the best\n",
    "approximation $u$ to $f$, with the given norm as measure of \"distance\".\n",
    "The second method is not\n",
    "as intuitive: we find $u$ such that the error $f-u$ is orthogonal to\n",
    "the space where $u$ lies. This is known as *projection*, or\n",
    "in the context of differential equations, the idea is\n",
    "also well known as *Galerkin's method*.\n",
    "When approximating vectors and functions, the two methods are\n",
    "equivalent, but this is no longer the case when applying the\n",
    "principles to differential equations.\n",
    "\n",
    "\n",
    "## Approximation of planar vectors\n",
    "<div id=\"fem:approx:vec:plane\"></div>\n",
    "\n",
    "Let $\\boldsymbol{f} = (3,5)$ be a vector in the $xy$ plane and suppose we want to\n",
    "approximate this vector by a vector aligned in the direction of\n",
    "another vector that is restricted to be aligned with some vector\n",
    "$(a,b)$. [Figure](#fem:approx:vec:plane:fig) depicts the\n",
    "situation. This is the simplest approximation problem for\n",
    "vectors. Nevertheless, for many readers it will be wise to refresh\n",
    "some basic linear algebra by consulting a textbook. Familiarity with fundamental operations on inner product vector spaces are assumed in the forthcoming text.\n",
    "\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/vecapprox_plane.png, width=400] Approximation of a two-dimensional vector in a one-dimensional vector space. <div id=\"fem:approx:vec:plane:fig\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:vec:plane:fig\"></div>\n",
    "\n",
    "<p>Approximation of a two-dimensional vector in a one-dimensional vector space.</p>\n",
    "<img src=\"fig/vecapprox_plane.png\" width=400>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "We introduce the vector space $V$\n",
    "spanned by the vector $\\boldsymbol{\\psi}_0=(a,b)$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto1\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "V = \\mbox{span}\\,\\{ \\boldsymbol{\\psi}_0\\}{\\thinspace .}   \\label{_auto1} \\tag{2}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We say that $\\boldsymbol{\\psi}_0$ is a *basis vector* in the space $V$.\n",
    "Our aim is to find the vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"uc0\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\label{uc0} \\tag{3}\n",
    "\\boldsymbol{u} = c_0\\boldsymbol{\\psi}_0\\in V\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which best approximates\n",
    "the given vector $\\boldsymbol{f} = (3,5)$. A reasonable criterion for a best\n",
    "approximation could be to minimize the length of the difference between\n",
    "the approximate $\\boldsymbol{u}$ and the given $\\boldsymbol{f}$. The difference, or error\n",
    "$\\boldsymbol{e} = \\boldsymbol{f} -\\boldsymbol{u}$, has its length given by the *norm*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "||\\boldsymbol{e}|| = (\\boldsymbol{e},\\boldsymbol{e})^{\\frac{1}{2}},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $(\\boldsymbol{e},\\boldsymbol{e})$ is the *inner product* of $\\boldsymbol{e}$ and itself. The inner\n",
    "product, also called *scalar product* or *dot product*, of two vectors\n",
    "$\\boldsymbol{u}=(u_0,u_1)$ and $\\boldsymbol{v} =(v_0,v_1)$ is defined as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto2\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{u}, \\boldsymbol{v}) = u_0v_0 + u_1v_1{\\thinspace .}   \\label{_auto2} \\tag{4}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark.** We should point out that we use the notation\n",
    "$(\\cdot,\\cdot)$ for two different things: $(a,b)$ for scalar\n",
    "quantities $a$ and $b$ means the vector starting at the origin and\n",
    "ending in the point $(a,b)$, while $(\\boldsymbol{u},\\boldsymbol{v})$ with vectors $\\boldsymbol{u}$ and\n",
    "$\\boldsymbol{v}$ means the inner product of these vectors.  Since vectors are here\n",
    "written in boldface font there should be no confusion.  We may add\n",
    "that the norm associated with this inner product is the usual\n",
    "Euclidean length of a vector, i.e.,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\|\\boldsymbol{u}\\| = \\sqrt{(\\boldsymbol{u},\\boldsymbol{u})} = \\sqrt{u_0^2 + u_1^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The least squares method\n",
    "\n",
    "We now want to determine the $\\boldsymbol{u}$ that minimizes  $||\\boldsymbol{e}||$, that is\n",
    "we want to compute the optimal $c_0$ in ([3](#uc0)). The algebra\n",
    "is simplified if we minimize the square of the norm, $||\\boldsymbol{e}||^2 = (\\boldsymbol{e}, \\boldsymbol{e})$,\n",
    "instead of the norm itself.\n",
    "Define the function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto3\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "E(c_0) = (\\boldsymbol{e},\\boldsymbol{e}) = (\\boldsymbol{f} - c_0\\boldsymbol{\\psi}_0, \\boldsymbol{f} - c_0\\boldsymbol{\\psi}_0)\n",
    "{\\thinspace .}\n",
    "\\label{_auto3} \\tag{5}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can rewrite the expressions of the right-hand side in a more\n",
    "convenient form for further use:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:E\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "E(c_0) = (\\boldsymbol{f},\\boldsymbol{f}) - 2c_0(\\boldsymbol{f},\\boldsymbol{\\psi}_0) + c_0^2(\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0){\\thinspace .}\n",
    "\\label{fem:vec:E} \\tag{6}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This rewrite results from using the following fundamental rules for inner\n",
    "product spaces:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:rule:scalarmult\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\alpha\\boldsymbol{u},\\boldsymbol{v})=\\alpha(\\boldsymbol{u},\\boldsymbol{v}),\\quad \\alpha\\in\\mathbb{R},\n",
    "\\label{fem:vec:rule:scalarmult} \\tag{7}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:rule:sum\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{u} +\\boldsymbol{v},\\boldsymbol{w}) = (\\boldsymbol{u},\\boldsymbol{w}) + (\\boldsymbol{v}, \\boldsymbol{w}),\n",
    "\\label{fem:vec:rule:sum} \\tag{8}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:rule:symmetry\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\label{fem:vec:rule:symmetry} \\tag{9}\n",
    "(\\boldsymbol{u}, \\boldsymbol{v}) = (\\boldsymbol{v}, \\boldsymbol{u}){\\thinspace .}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Minimizing $E(c_0)$ implies finding $c_0$ such that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial E}{\\partial c_0} = 0{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out that $E$ has one unique minimum and no maximum point.\n",
    "Now, when differentiating ([6](#fem:vec:E)) with respect to $c_0$, note\n",
    "that none of the inner product expressions depend on $c_0$, so we simply get"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:dEdc0:v1\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{\\partial E}{\\partial c_0} = -2(\\boldsymbol{f},\\boldsymbol{\\psi}_0) + 2c_0 (\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0)\n",
    "{\\thinspace .}\n",
    "\\label{fem:vec:dEdc0:v1} \\tag{10}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting the above expression equal to zero and solving for $c_0$ gives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:c0\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "c_0 = \\frac{(\\boldsymbol{f},\\boldsymbol{\\psi}_0)}{(\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0)},\n",
    "\\label{fem:vec:c0} \\tag{11}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which in the present case, with $\\boldsymbol{\\psi}_0=(a,b)$, results in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto4\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "c_0 = \\frac{3a + 5b}{a^2 + b^2}{\\thinspace .}   \\label{_auto4} \\tag{12}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For later, it is worth mentioning that setting\n",
    "the key equation ([10](#fem:vec:dEdc0:v1)) to zero and ordering\n",
    "the terms lead to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(\\boldsymbol{f}-c_0\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0) = 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:dEdc0:Galerkin\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{e}, \\boldsymbol{\\psi}_0) = 0\n",
    "{\\thinspace .}\n",
    "\\label{fem:vec:dEdc0:Galerkin} \\tag{13}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This implication of minimizing $E$ is an important result that we shall\n",
    "make much use of.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### The projection method\n",
    "\n",
    "We shall now show that minimizing $||\\boldsymbol{e}||^2$ implies that $\\boldsymbol{e}$ is\n",
    "orthogonal to *any* vector $\\boldsymbol{v}$ in the space $V$. This result is\n",
    "visually quite clear from [Figure](#fem:approx:vec:plane:fig) (think of\n",
    "other vectors along the line $(a,b)$: all of them will lead to\n",
    "a larger distance between the approximation and $\\boldsymbol{f}$).\n",
    "Then we see mathematically that $\\boldsymbol{e}$ is orthogonal to any vector $\\boldsymbol{v}$\n",
    "in the space $V$ and we may\n",
    "express any $\\boldsymbol{v}\\in V$ as $\\boldsymbol{v}=s\\boldsymbol{\\psi}_0$ for any scalar parameter $s$\n",
    "(recall that two vectors are orthogonal when their inner product vanishes).\n",
    "Then we calculate the inner product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "(\\boldsymbol{e}, s\\boldsymbol{\\psi}_0) &= (\\boldsymbol{f} - c_0\\boldsymbol{\\psi}_0, s\\boldsymbol{\\psi}_0)\\\\\n",
    "&= (\\boldsymbol{f},s\\boldsymbol{\\psi}_0) - (c_0\\boldsymbol{\\psi}_0, s\\boldsymbol{\\psi}_0)\\\\\n",
    "&= s(\\boldsymbol{f},\\boldsymbol{\\psi}_0) - sc_0(\\boldsymbol{\\psi}_0, \\boldsymbol{\\psi}_0)\\\\\n",
    "&= s(\\boldsymbol{f},\\boldsymbol{\\psi}_0) - s\\frac{(\\boldsymbol{f},\\boldsymbol{\\psi}_0)}{(\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0)}(\\boldsymbol{\\psi}_0,\\boldsymbol{\\psi}_0)\\\\\n",
    "&= s\\left( (\\boldsymbol{f},\\boldsymbol{\\psi}_0) - (\\boldsymbol{f},\\boldsymbol{\\psi}_0)\\right)\\\\\n",
    "&=0{\\thinspace .}\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, instead of minimizing the square of the norm, we could\n",
    "demand that $\\boldsymbol{e}$ is orthogonal to any vector in $V$, which in our present\n",
    "simple case amounts to a single vector only.\n",
    "This method is known as *projection*.\n",
    "(The approach can also be referred to as a Galerkin method as\n",
    "explained at the end of the section [Approximation of general vectors](#fem:approx:vec:Np1dim).)\n",
    "\n",
    "Mathematically, the projection method is stated\n",
    "by the equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:Galerkin1\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{e}, \\boldsymbol{v}) = 0,\\quad\\forall\\boldsymbol{v}\\in V{\\thinspace .}\n",
    "\\label{fem:vec:Galerkin1} \\tag{14}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An arbitrary $\\boldsymbol{v}\\in V$ can be expressed as\n",
    "$s\\boldsymbol{\\psi}_0$, $s\\in\\mathbb{R}$, and therefore\n",
    "([14](#fem:vec:Galerkin1)) implies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(\\boldsymbol{e},s\\boldsymbol{\\psi}_0) = s(\\boldsymbol{e}, \\boldsymbol{\\psi}_0) = 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which means that the error must be orthogonal to the basis vector in\n",
    "the space $V$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(\\boldsymbol{e}, \\boldsymbol{\\psi}_0)=0\\quad\\hbox{or}\\quad\n",
    "(\\boldsymbol{f} - c_0\\boldsymbol{\\psi}_0, \\boldsymbol{\\psi}_0)=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which is what we found in ([13](#fem:vec:dEdc0:Galerkin)) from\n",
    "the least squares computations.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Approximation of general vectors\n",
    "<div id=\"fem:approx:vec:Np1dim\"></div>\n",
    "\n",
    "\n",
    "Let us generalize the vector approximation from the previous section\n",
    "to vectors in spaces with arbitrary dimension. Given some vector $\\boldsymbol{f}$,\n",
    "we want to find the best approximation to this vector in\n",
    "the space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "V = \\hbox{span}\\,\\{\\boldsymbol{\\psi}_0,\\ldots,\\boldsymbol{\\psi}_N\\}\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume that the space has dimension $N+1$ and\n",
    "that *basis vectors* $\\boldsymbol{\\psi}_0,\\ldots,\\boldsymbol{\\psi}_N$ are\n",
    "linearly independent so that none of them are redundant.\n",
    "Any vector $\\boldsymbol{u}\\in V$ can then be written as a linear combination\n",
    "of the basis vectors, i.e.,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\boldsymbol{u} = \\sum_{j=0}^N c_j\\boldsymbol{\\psi}_j,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $c_j\\in\\mathbb{R}$ are scalar coefficients to be determined.\n",
    "\n",
    "### The least squares method\n",
    "\n",
    "Now we want to find $c_0,\\ldots,c_N$, such that $\\boldsymbol{u}$ is the best\n",
    "approximation to $\\boldsymbol{f}$ in the sense that the distance (error)\n",
    "$\\boldsymbol{e} = \\boldsymbol{f} - \\boldsymbol{u}$ is minimized. Again, we define\n",
    "the squared distance as a function of the free parameters\n",
    "$c_0,\\ldots,c_N$,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "E(c_0,\\ldots,c_N) = (\\boldsymbol{e},\\boldsymbol{e}) = (\\boldsymbol{f} -\\sum_jc_j\\boldsymbol{\\psi}_j,\\boldsymbol{f} -\\sum_jc_j\\boldsymbol{\\psi}_j)\n",
    "\\nonumber\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:vec:genE\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "= (\\boldsymbol{f},\\boldsymbol{f}) - 2\\sum_{j=0}^N c_j(\\boldsymbol{f},\\boldsymbol{\\psi}_j) +\n",
    "\\sum_{p=0}^N\\sum_{q=0}^N c_pc_q(\\boldsymbol{\\psi}_p,\\boldsymbol{\\psi}_q){\\thinspace .}\n",
    "\\label{fem:vec:genE} \\tag{15}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Minimizing this $E$ with respect to the independent variables\n",
    "$c_0,\\ldots,c_N$ is obtained by requiring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial E}{\\partial c_i} = 0,\\quad i=0,\\ldots,N\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first term in ([15](#fem:vec:genE)) is independent of $c_i$, so its\n",
    "derivative vanishes.\n",
    "The second term in ([15](#fem:vec:genE)) is differentiated as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto5\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{\\partial}{\\partial c_i}\n",
    "2\\sum_{j=0}^N c_j(\\boldsymbol{f},\\boldsymbol{\\psi}_j) = 2(\\boldsymbol{f},\\boldsymbol{\\psi}_i),\n",
    "\\label{_auto5} \\tag{16}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since the expression to be differentiated is a sum and only one term,\n",
    "$c_i(\\boldsymbol{f},\\boldsymbol{\\psi}_i)$,\n",
    "contains $c_i$ (this term is linear in $c_i$).\n",
    "To understand this differentiation in detail,\n",
    "write out the sum specifically for,\n",
    "e.g, $N=3$ and $i=1$.\n",
    "\n",
    "The last term in ([15](#fem:vec:genE))\n",
    "is more tedious to differentiate. It can be wise to write out the\n",
    "double sum for $N=1$ and perform differentiation with respect to\n",
    "$c_0$ and $c_1$ to see the structure of the expression. Thereafter,\n",
    "one can generalize to an arbitrary $N$ and observe that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto6\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{\\partial}{\\partial c_i}\n",
    "c_pc_q =\n",
    "\\left\\lbrace\\begin{array}{ll}\n",
    "0, & \\hbox{ if } p\\neq i\\hbox{ and } q\\neq i,\\\\\n",
    "c_q, & \\hbox{ if } p=i\\hbox{ and } q\\neq i,\\\\\n",
    "c_p, & \\hbox{ if } p\\neq i\\hbox{ and } q=i,\\\\\n",
    "2c_i, & \\hbox{ if } p=q= i{\\thinspace .}\\\\\n",
    "\\end{array}\\right.\n",
    "\\label{_auto6} \\tag{17}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial}{\\partial c_i}\n",
    "\\sum_{p=0}^N\\sum_{q=0}^N c_pc_q(\\boldsymbol{\\psi}_p,\\boldsymbol{\\psi}_q)\n",
    "= \\sum_{p=0, p\\neq i}^N c_p(\\boldsymbol{\\psi}_p,\\boldsymbol{\\psi}_i)\n",
    "+ \\sum_{q=0, q\\neq i}^N c_q(\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_q)\n",
    "+2c_i(\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_i){\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each of the two sums is missing the term $c_i(\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_i)$,\n",
    "we may split the very last term in two, to get exactly that \"missing\"\n",
    "term for each sum. This idea allows us to write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto7\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{\\partial}{\\partial c_i}\n",
    "\\sum_{p=0}^N\\sum_{q=0}^N c_pc_q(\\boldsymbol{\\psi}_p,\\boldsymbol{\\psi}_q)\n",
    "= 2\\sum_{j=0}^N c_i(\\boldsymbol{\\psi}_j,\\boldsymbol{\\psi}_i){\\thinspace .}\n",
    "\\label{_auto7} \\tag{18}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It then follows that setting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial E}{\\partial c_i} = 0,\\quad i=0,\\ldots,N,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "implies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "- 2(\\boldsymbol{f},\\boldsymbol{\\psi}_i) + 2\\sum_{j=0}^N c_i(\\boldsymbol{\\psi}_j,\\boldsymbol{\\psi}_i) = 0,\\quad i=0,\\ldots,N{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving the first term to the right-hand side shows that the equation is\n",
    "actually a *linear system* for the unknown parameters $c_0,\\ldots,c_N$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:vec:Np1dim:eqsys\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\sum_{j=0}^N A_{i,j} c_j = b_i, \\quad i=0,\\ldots,N,\n",
    "\\label{fem:approx:vec:Np1dim:eqsys} \\tag{19}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto8\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "A_{i,j} = (\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_j),\n",
    "\\label{_auto8} \\tag{20}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto9\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "b_i = (\\boldsymbol{\\psi}_i, \\boldsymbol{f}){\\thinspace .}   \\label{_auto9} \\tag{21}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have changed the order of the two vectors in the inner\n",
    "product according to ([9](#fem:vec:rule:symmetry)):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "A_{i,j} = (\\boldsymbol{\\psi}_j,\\boldsymbol{\\psi}_i) = (\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_j),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "simply because the sequence $i$-$j$ looks more aesthetic.\n",
    "\n",
    "### The Galerkin or projection method\n",
    "\n",
    "In analogy with the \"one-dimensional\" example in\n",
    "the section [Approximation of planar vectors](#fem:approx:vec:plane), it holds also here in the general\n",
    "case that minimizing the distance\n",
    "(error) $\\boldsymbol{e}$ is equivalent to demanding that $\\boldsymbol{e}$ is orthogonal to\n",
    "all $\\boldsymbol{v}\\in V$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:vec:Np1dim:Galerkin\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{e},\\boldsymbol{v})=0,\\quad \\forall\\boldsymbol{v}\\in V{\\thinspace .}\n",
    "\\label{fem:approx:vec:Np1dim:Galerkin} \\tag{22}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since any $\\boldsymbol{v}\\in V$ can be written as $\\boldsymbol{v} =\\sum_{i=0}^N c_i\\boldsymbol{\\psi}_i$,\n",
    "the statement ([22](#fem:approx:vec:Np1dim:Galerkin)) is equivalent to\n",
    "saying that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(\\boldsymbol{e}, \\sum_{i=0}^N c_i\\boldsymbol{\\psi}_i) = 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for any choice of coefficients $c_0,\\ldots,c_N$.\n",
    "The latter equation can be rewritten as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sum_{i=0}^N c_i (\\boldsymbol{e},\\boldsymbol{\\psi}_i) =0{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this is to hold for arbitrary values of $c_0,\\ldots,c_N$,\n",
    "we must require that each term in the sum vanishes, which means that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:vec:Np1dim:Galerkin0\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(\\boldsymbol{e},\\boldsymbol{\\psi}_i)=0,\\quad i=0,\\ldots,N{\\thinspace .}\n",
    "\\label{fem:approx:vec:Np1dim:Galerkin0} \\tag{23}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These $N+1$ equations result in the same linear system as\n",
    "([19](#fem:approx:vec:Np1dim:eqsys)):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(\\boldsymbol{f} - \\sum_{j=0}^N c_j\\boldsymbol{\\psi}_j, \\boldsymbol{\\psi}_i) = (\\boldsymbol{f}, \\boldsymbol{\\psi}_i) - \\sum_{j=0}^N\n",
    "(\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_j)c_j = 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and hence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sum_{j=0}^N (\\boldsymbol{\\psi}_i,\\boldsymbol{\\psi}_j)c_j = (\\boldsymbol{f}, \\boldsymbol{\\psi}_i),\\quad i=0,\\ldots, N\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, instead of differentiating the\n",
    "$E(c_0,\\ldots,c_N)$ function, we could simply use\n",
    "([22](#fem:approx:vec:Np1dim:Galerkin)) as the principle for\n",
    "determining $c_0,\\ldots,c_N$, resulting in the $N+1$\n",
    "equations ([23](#fem:approx:vec:Np1dim:Galerkin0)).\n",
    "\n",
    "The names *least squares method* or *least squares approximation*\n",
    "are natural since the calculations consists of\n",
    "minimizing $||\\boldsymbol{e}||^2$, and $||\\boldsymbol{e}||^2$ is a sum of squares\n",
    "of differences between the components in $\\boldsymbol{f}$ and $\\boldsymbol{u}$.\n",
    "We find $\\boldsymbol{u}$ such that this sum of squares is minimized.\n",
    "\n",
    "The principle ([22](#fem:approx:vec:Np1dim:Galerkin)),\n",
    "or the equivalent form ([23](#fem:approx:vec:Np1dim:Galerkin0)),\n",
    "is known as *projection*. Almost the same mathematical idea\n",
    "was used by the Russian mathematician [Boris Galerkin](http://en.wikipedia.org/wiki/Boris_Galerkin) to solve\n",
    "differential equations, resulting in what is widely known as\n",
    "*Galerkin's method*.\n",
    "\n",
    "# Approximation principles\n",
    "<div id=\"fem:approx:global\"></div>\n",
    "\n",
    "Let $V$ be a function space spanned by a set of *basis functions*\n",
    "${\\psi}_0,\\ldots,{\\psi}_N$,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "V = \\hbox{span}\\,\\{{\\psi}_0,\\ldots,{\\psi}_N\\},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "such that any function $u\\in V$ can be written as a linear\n",
    "combination of the basis functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:ufem\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\label{fem:approx:ufem} \\tag{24}\n",
    "u = \\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j{\\thinspace .}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, we consider functions as vectors in a vector space - a\n",
    "so-called function space - and we have a finite set of basis\n",
    "functions that span the space just as basis vectors or unit vectors\n",
    "span a vector space.\n",
    "\n",
    "The index set ${\\mathcal{I}_s}$ is defined as ${\\mathcal{I}_s} =\\{0,\\ldots,N\\}$ and is\n",
    "from now on used\n",
    "both for compact notation and for flexibility in the numbering of\n",
    "elements in sequences.\n",
    "\n",
    "For now, in this introduction, we shall look at functions of a\n",
    "single variable $x$:\n",
    "$u=u(x)$, ${\\psi}_j={\\psi}_j(x)$, $j\\in{\\mathcal{I}_s}$. Later, we will almost\n",
    "trivially extend the mathematical details\n",
    "to functions of two- or three-dimensional physical spaces.\n",
    "The approximation ([24](#fem:approx:ufem)) is typically used\n",
    "to discretize a problem in space. Other methods, most notably\n",
    "finite differences, are common for time discretization, although the\n",
    "form ([24](#fem:approx:ufem)) can be used in time as well.\n",
    "\n",
    "## The least squares method\n",
    "<div id=\"fem:approx:LS\"></div>\n",
    "\n",
    "Given a function $f(x)$, how can we determine its best approximation\n",
    "$u(x)\\in V$? A natural starting point is to apply the same reasoning\n",
    "as we did for vectors in the section [Approximation of general vectors](#fem:approx:vec:Np1dim). That is,\n",
    "we minimize the distance between $u$ and $f$. However, this requires\n",
    "a norm for measuring distances, and a norm is most conveniently\n",
    "defined through an\n",
    "inner product. Viewing a function as a vector of infinitely\n",
    "many point values, one for each value of $x$, the inner product of\n",
    "two arbitrary functions $f(x)$ and $g(x)$ could\n",
    "intuitively be defined as the usual summation of\n",
    "pairwise \"components\" (values), with summation replaced by integration:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "(f,g) = \\int f(x)g(x)\\, {\\, \\mathrm{d}x}\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fix the integration domain, we let $f(x)$ and ${\\psi}_i(x)$\n",
    "be defined for a domain $\\Omega\\subset\\mathbb{R}$.\n",
    "The inner product of two functions $f(x)$ and $g(x)$ is then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:LS:innerprod\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(f,g) = \\int_\\Omega f(x)g(x)\\, {\\, \\mathrm{d}x}\n",
    "\\label{fem:approx:LS:innerprod} \\tag{25}\n",
    "{\\thinspace .}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distance between $f$ and any function $u\\in V$ is simply\n",
    "$f-u$, and the squared norm of this distance is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:LS:E\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "E = (f(x)-\\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x), f(x)-\\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x)){\\thinspace .}\n",
    "\\label{fem:approx:LS:E} \\tag{26}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the analogy with ([15](#fem:vec:genE)): the given function\n",
    "$f$ plays the role of the given vector $\\boldsymbol{f}$, and the basis function\n",
    "${\\psi}_i$ plays the role of the basis vector $\\boldsymbol{\\psi}_i$.\n",
    "We can rewrite ([26](#fem:approx:LS:E)),\n",
    "through similar steps as used for the result\n",
    "([15](#fem:vec:genE)), leading to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto10\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "E(c_i, \\ldots, c_N) = (f,f) -2\\sum_{j\\in{\\mathcal{I}_s}} c_j(f,{\\psi}_i)\n",
    "+ \\sum_{p\\in{\\mathcal{I}_s}}\\sum_{q\\in{\\mathcal{I}_s}} c_pc_q({\\psi}_p,{\\psi}_q){\\thinspace .}   \\label{_auto10} \\tag{27}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Minimizing this function of $N+1$ scalar variables\n",
    "$\\left\\{ {c}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$, requires differentiation\n",
    "with respect to $c_i$, for all $i\\in{\\mathcal{I}_s}$. The resulting\n",
    "equations are very similar to those we had in the vector case,\n",
    "and we hence end up with a\n",
    "linear system of the form ([19](#fem:approx:vec:Np1dim:eqsys)), with\n",
    "basically the same expressions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:Aij\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "A_{i,j} = ({\\psi}_i,{\\psi}_j),\n",
    "\\label{fem:approx:Aij} \\tag{28}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:bi\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "b_i = (f,{\\psi}_i){\\thinspace .}\n",
    "\\label{fem:approx:bi} \\tag{29}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only difference from\n",
    "([19](#fem:approx:vec:Np1dim:eqsys))\n",
    "is that the inner product is defined in terms\n",
    "of integration rather than summation.\n",
    "\n",
    "## The projection (or Galerkin) method\n",
    "\n",
    "\n",
    "As in the section [Approximation of general vectors](#fem:approx:vec:Np1dim), the minimization of $(e,e)$\n",
    "is equivalent to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:Galerkin\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(e,v)=0,\\quad\\forall v\\in V{\\thinspace .}\n",
    "\\label{fem:approx:Galerkin} \\tag{30}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is known as a projection of a function $f$ onto the subspace $V$.\n",
    "We may also call it a Galerkin method for approximating functions.\n",
    "Using the same reasoning as\n",
    "in\n",
    "([22](#fem:approx:vec:Np1dim:Galerkin))-([23](#fem:approx:vec:Np1dim:Galerkin0)),\n",
    "it follows that ([30](#fem:approx:Galerkin)) is equivalent to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:Galerkin0\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(e,{\\psi}_i)=0,\\quad i\\in{\\mathcal{I}_s}{\\thinspace .}\n",
    "\\label{fem:approx:Galerkin0} \\tag{31}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inserting $e=f-u$ in this equation and ordering terms, as in the\n",
    "multi-dimensional vector case, we end up with a linear\n",
    "system with a coefficient matrix ([28](#fem:approx:Aij)) and\n",
    "right-hand side vector ([29](#fem:approx:bi)).\n",
    "\n",
    "Whether we work with vectors in the plane, general vectors, or\n",
    "functions in function spaces, the least squares principle and\n",
    "the projection or Galerkin method are equivalent.\n",
    "\n",
    "## Example of linear approximation\n",
    "<div id=\"fem:approx:global:linear\"></div>\n",
    "\n",
    "Let us apply the theory in the previous section to a simple problem:\n",
    "given a parabola $f(x)=10(x-1)^2-1$ for $x\\in\\Omega=[1,2]$, find\n",
    "the best approximation $u(x)$ in the space of all linear functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "V = \\hbox{span}\\,\\{1, x\\}{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our notation, ${\\psi}_0(x)=1$, ${\\psi}_1(x)=x$, and $N=1$.\n",
    "We seek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "u=c_0{\\psi}_0(x) + c_1{\\psi}_1(x) = c_0 + c_1x,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where\n",
    "$c_0$ and $c_1$ are found by solving a $2\\times 2$ linear system.\n",
    "The coefficient matrix has elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto11\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "A_{0,0} = ({\\psi}_0,{\\psi}_0) = \\int_1^21\\cdot 1\\, {\\, \\mathrm{d}x} = 1,\n",
    "\\label{_auto11} \\tag{32}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto12\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "A_{0,1} = ({\\psi}_0,{\\psi}_1) = \\int_1^2 1\\cdot x\\, {\\, \\mathrm{d}x} = 3/2,\n",
    "\\label{_auto12} \\tag{33}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto13\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "A_{1,0} = A_{0,1} = 3/2,\n",
    "\\label{_auto13} \\tag{34}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto14\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "A_{1,1} = ({\\psi}_1,{\\psi}_1) = \\int_1^2 x\\cdot x\\,{\\, \\mathrm{d}x} = 7/3{\\thinspace .}   \\label{_auto14} \\tag{35}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corresponding right-hand side is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto15\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "b_1 = (f,{\\psi}_0) = \\int_1^2 (10(x-1)^2 - 1)\\cdot 1 \\, {\\, \\mathrm{d}x} = 7/3,\n",
    "\\label{_auto15} \\tag{36}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto16\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "b_2 = (f,{\\psi}_1) = \\int_1^2 (10(x-1)^2 - 1)\\cdot x\\, {\\, \\mathrm{d}x} = 13/3{\\thinspace .}   \\label{_auto16} \\tag{37}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solving the linear system results in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto17\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "c_0 = -38/3,\\quad c_1 = 10,\n",
    "\\label{_auto17} \\tag{38}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and consequently"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto18\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "u(x) = 10x - \\frac{38}{3}{\\thinspace .}   \\label{_auto18} \\tag{39}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Figure](#fem:approx:global:fig:parabola:linear) displays the\n",
    "parabola and its best approximation in the space of all linear functions.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/parabola_ls_linear.png, width=400]  Best approximation of a parabola by a straight line.  <div id=\"fem:approx:global:fig:parabola:linear\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:fig:parabola:linear\"></div>\n",
    "\n",
    "<p>Best approximation of a parabola by a straight line.</p>\n",
    "<img src=\"fig/parabola_ls_linear.png\" width=400>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "## Implementation of the least squares method\n",
    "<div id=\"fem:approx:global:LS:code\"></div>\n",
    "\n",
    "### Symbolic integration\n",
    "\n",
    "The linear system can be computed either symbolically or\n",
    "numerically (a numerical integration rule is needed in the latter case).\n",
    "Let us first compute the system and its solution symbolically, i.e.,\n",
    "using classical \"pen and paper\" mathematics with symbols.\n",
    "The Python package `sympy` can greatly help with this type of\n",
    "mathematics, and will therefore be frequently used in this text.\n",
    "Some basic familiarity with `sympy` is assumed, typically\n",
    "`symbols`, `integrate`, `diff`, `expand`, and `simplify`. Much can be learned\n",
    "by studying the many applications of `sympy` that will be presented.\n",
    "\n",
    "Below is a function for symbolic computation of the linear system,\n",
    "where $f(x)$ is given as a `sympy` expression `f` involving\n",
    "the symbol `x`, `psi` is a list of expressions for $\\left\\{ {{\\psi}}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$,\n",
    "and `Omega` is a 2-tuple/list holding the limits of the domain $\\Omega$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sympy as sym\n",
    "\n",
    "def least_squares(f, psi, Omega):\n",
    "    N = len(psi) - 1\n",
    "    A = sym.zeros(N+1, N+1)\n",
    "    b = sym.zeros(N+1, 1)\n",
    "    x = sym.Symbol('x')\n",
    "    for i in range(N+1):\n",
    "        for j in range(i, N+1):\n",
    "            A[i,j] = sym.integrate(psi[i]*psi[j],\n",
    "                                  (x, Omega[0], Omega[1]))\n",
    "            A[j,i] = A[i,j]\n",
    "        b[i,0] = sym.integrate(psi[i]*f, (x, Omega[0], Omega[1]))\n",
    "    c = A.LUsolve(b)\n",
    "    # Note: c is a sympy Matrix object, solution is in c[:,0]\n",
    "    u = 0\n",
    "    for i in range(len(psi)):\n",
    "        u += c[i,0]*psi[i]\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe that we exploit the symmetry of the coefficient matrix:\n",
    "only the upper triangular part is computed. Symbolic integration, also in\n",
    "`sympy`, is often time consuming, and (roughly) halving the\n",
    "work has noticeable effect on the waiting time for the computations to\n",
    "finish.\n",
    "\n",
    "**Notice.**\n",
    "\n",
    "We remark that the symbols in `sympy` are created and stored in\n",
    "a symbol factory that is indexed by the expression used in the construction\n",
    "and that repeated constructions from the same expression will not create\n",
    "new objects. The following code illustrates the behavior of the\n",
    "symbol factory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sympy import *\n",
    "x0 = Symbol(\"x\")\n",
    "x1 = Symbol(\"x\")\n",
    "id(x0) ==id(x1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a0 = 3.0\n",
    "a1 = 3.0\n",
    "id(a0) ==id(a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fall back on numerical integration\n",
    "\n",
    "Obviously, `sympy` may fail to successfully integrate\n",
    "$\\int_\\Omega{\\psi}_i{\\psi}_j{\\, \\mathrm{d}x}$, and\n",
    "especially $\\int_\\Omega f{\\psi}_i{\\, \\mathrm{d}x}$, symbolically.\n",
    "Therefore, we should extend\n",
    "the `least_squares` function such that it falls back on\n",
    "numerical integration if the symbolic integration is unsuccessful.\n",
    "In the latter case, the returned value from `sympy`'s\n",
    "`integrate` function is an object of type `Integral`.\n",
    "We can test on this type and utilize the `mpmath` module\n",
    "to perform numerical integration of high precision.\n",
    "Even when `sympy` manages to integrate symbolically, it can\n",
    "take an undesirably long time. We therefore include an\n",
    "argument `symbolic` that governs whether or not to try\n",
    "symbolic integration. Here is a complete and\n",
    "improved version of the previous function `least_squares`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares(f, psi, Omega, symbolic=True):\n",
    "    N = len(psi) - 1\n",
    "    A = sym.zeros(N+1, N+1)\n",
    "    b = sym.zeros(N+1, 1)\n",
    "    x = sym.Symbol('x')\n",
    "    for i in range(N+1):\n",
    "        for j in range(i, N+1):\n",
    "            integrand = psi[i]*psi[j]\n",
    "            if symbolic:\n",
    "                I = sym.integrate(integrand, (x, Omega[0], Omega[1]))\n",
    "            if not symbolic or isinstance(I, sym.Integral):\n",
    "                # Could not integrate symbolically, use numerical int.\n",
    "                integrand = sym.lambdify([x], integrand, 'mpmath')\n",
    "                I = mpmath.quad(integrand, [Omega[0], Omega[1]])\n",
    "            A[i,j] = A[j,i] = I\n",
    "\n",
    "        integrand = psi[i]*f\n",
    "        if symbolic:\n",
    "            I = sym.integrate(integrand, (x, Omega[0], Omega[1]))\n",
    "        if not symbolic or isinstance(I, sym.Integral):\n",
    "            # Could not integrate symbolically, use numerical int.\n",
    "            integrand = sym.lambdify([x], integrand, 'mpmath')\n",
    "            I = mpmath.quad(integrand, [Omega[0], Omega[1]])\n",
    "        b[i,0] = I\n",
    "    if symbolic:\n",
    "        c = A.LUsolve(b)  # symbolic solve\n",
    "        # c is a sympy Matrix object, numbers are in c[i,0]\n",
    "        c = [sym.simplify(c[i,0]) for i in range(c.shape[0])]\n",
    "    else:\n",
    "        c = mpmath.lu_solve(A, b)  # numerical solve\n",
    "        c = [c[i,0] for i in range(c.rows)]\n",
    "    u = sum(c[i]*psi[i] for i in range(len(psi)))\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function is found in the file `approx1D.py`.\n",
    "\n",
    "### Plotting the approximation\n",
    "\n",
    "Comparing the given $f(x)$ and the approximate $u(x)$ visually is done\n",
    "with the following function, which utilizes `sympy`'s `lambdify` tool to\n",
    "convert a `sympy` expression to a Python function for numerical\n",
    "computations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def comparison_plot(f, u, Omega, filename='tmp.pdf'):\n",
    "    x = sym.Symbol('x')\n",
    "    f = sym.lambdify([x], f, modules=\"numpy\")\n",
    "    u = sym.lambdify([x], u, modules=\"numpy\")\n",
    "    resolution = 401  # no of points in plot\n",
    "    xcoor  = np.linspace(Omega[0], Omega[1], resolution)\n",
    "    exact  = f(xcoor)\n",
    "    approx = u(xcoor)\n",
    "    plt.plot(xcoor, approx)\n",
    "    plt.plot(xcoor, exact)\n",
    "    plt.legend(['approximation', 'exact'])\n",
    "    # plt.savefig(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `modules='numpy'` argument to `lambdify` is important\n",
    "if there are mathematical functions, such as `sin` or `exp`\n",
    "in the symbolic expressions in `f` or `u`, and these\n",
    "mathematical functions are to be used with vector arguments, like\n",
    "`xcoor` above.\n",
    "\n",
    "Both the `least_squares` and `comparison_plot` functions are found in\n",
    "the file [`approx1D.py`](src/approx1D.py).  The\n",
    "`comparison_plot` function in this file is more advanced and flexible\n",
    "than the simplistic version shown above.  The file [`ex_approx1D.py`](src/ex_approx1D.py)\n",
    "applies the `approx1D` module to accomplish the forthcoming examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perfect approximation\n",
    "<div id=\"fem:approx:global:exact1\"></div>\n",
    "\n",
    "Let us use the code above to recompute the problem from\n",
    "the section [Example of linear approximation](#fem:approx:global:linear) where we want to approximate\n",
    "a parabola. What happens if we add an element $x^2$ to the set of basis functions and test what\n",
    "the best approximation is if $V$ is the space of all parabolic functions?\n",
    "The answer is quickly found by running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10*x**2 - 20*x + 9\n",
      "10*x**2 - 20*x + 9\n"
     ]
    }
   ],
   "source": [
    "x = sym.Symbol('x')\n",
    "f = 10*(x-1)**2-1\n",
    "u, c = least_squares(f=f, psi=[1, x, x**2], Omega=[1, 2])\n",
    "print(u)\n",
    "print(sym.expand(f))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, what if we use ${\\psi}_i(x)=x^i$ for $i=0,1,\\ldots,N=40$?\n",
    "The output from `least_squares` gives $c_i=0$ for $i>2$, which\n",
    "means that the method finds the perfect approximation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The residual: an indirect but computationally cheap measure of the error.\n",
    "\n",
    "When attempting to  solve\n",
    "a system $A c = b$, we may question how far off a start vector or a current approximation $c_0$ is.\n",
    "The error is clearly the difference between $c$ and $c_0$, $e=c-c_0$, but since we do\n",
    "not know the true solution $c$ we are unable to assess the error.\n",
    "However, the vector $c_0$ is the solution of the an alternative problem $A c_0 = b_0$.\n",
    "If the input, i.e., the right-hand sides $b_0$ and $b$ are close to each other then\n",
    "we expect the output of a solution process $c$ and $c_0$ to be close to each other.\n",
    "Furthermore, while $b_0$ in principle is unknown, it is easily computable as $b_0 = A c_0$\n",
    "and does not require inversion of $A$.\n",
    "The vector $b - b_0$ is the so-called residual $r$ defined by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "r = b - b_0 = b - A c_0 =  A c - A c_0 .\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, the error and the residual are related by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "A e = r .\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the computation of the error requires inversion of $A$,\n",
    "which may be computationally expensive, the residual\n",
    "is easily computable and do only require a matrix-vector product and vector additions.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Orthogonal basis functions\n",
    "\n",
    "Approximation of a function via orthogonal functions, especially sinusoidal\n",
    "functions or orthogonal polynomials, is a very popular and successful approach.\n",
    "The finite element method does not make use of orthogonal functions, but\n",
    "functions that are \"almost orthogonal\".\n",
    "\n",
    "\n",
    "## Ill-conditioning\n",
    "<div id=\"fem:approx:global:illconditioning\"></div>\n",
    "\n",
    "For basis functions that are not orthogonal the condition number\n",
    "of the matrix may create problems during the solution process\n",
    "due to, for example, round-off errors as will be illustrated in the\n",
    "following.   The computational example in the section [Perfect approximation](#fem:approx:global:exact1)\n",
    "applies the `least_squares` function which invokes symbolic\n",
    "methods to calculate and solve the linear system. The correct\n",
    "solution $c_0=9, c_1=-20, c_2=10, c_i=0$ for $i\\geq 3$ is perfectly\n",
    "recovered.\n",
    "\n",
    "Suppose we\n",
    "convert the matrix and right-hand side to floating-point arrays\n",
    "and then solve the system using finite-precision arithmetics, which\n",
    "is what one will (almost) always do in real life. This time we\n",
    "get astonishing results! Up to about $N=7$ we get a solution that\n",
    "is reasonably close to the exact one. Increasing $N$ shows that\n",
    "seriously wrong coefficients are computed.\n",
    "Below is a table showing the solution of the linear system arising from\n",
    "approximating a parabola\n",
    "by functions of the form $u(x)=c_0 + c_1x + c_2x^2 + \\cdots + c_{10}x^{10}$.\n",
    "Analytically, we know that $c_j=0$ for $j>2$, but numerically we may get\n",
    "$c_j\\neq 0$ for $j>2$.\n",
    "\n",
    "<table border=\"1\">\n",
    "<thead>\n",
    "<tr><th align=\"center\">exact</th> <th align=\"center\"><code>sympy</code></th> <th align=\"center\"><code>numpy32</code></th> <th align=\"center\"><code>numpy64</code></th> </tr>\n",
    "</thead>\n",
    "<tbody>\n",
    "<tr><td align=\"right\">   9        </td> <td align=\"right\">   9.62       </td> <td align=\"right\">   5.57         </td> <td align=\"right\">   8.98         </td> </tr>\n",
    "<tr><td align=\"right\">   -20      </td> <td align=\"right\">   -23.39     </td> <td align=\"right\">   -7.65        </td> <td align=\"right\">   -19.93       </td> </tr>\n",
    "<tr><td align=\"right\">   10       </td> <td align=\"right\">   17.74      </td> <td align=\"right\">   -4.50        </td> <td align=\"right\">   9.96         </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   -9.19      </td> <td align=\"right\">   4.13         </td> <td align=\"right\">   -0.26        </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   5.25       </td> <td align=\"right\">   2.99         </td> <td align=\"right\">   0.72         </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   0.18       </td> <td align=\"right\">   -1.21        </td> <td align=\"right\">   -0.93        </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   -2.48      </td> <td align=\"right\">   -0.41        </td> <td align=\"right\">   0.73         </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   1.81       </td> <td align=\"right\">   -0.013       </td> <td align=\"right\">   -0.36        </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   -0.66      </td> <td align=\"right\">   0.08         </td> <td align=\"right\">   0.11         </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   0.12       </td> <td align=\"right\">   0.04         </td> <td align=\"right\">   -0.02        </td> </tr>\n",
    "<tr><td align=\"right\">   0        </td> <td align=\"right\">   -0.001     </td> <td align=\"right\">   -0.02        </td> <td align=\"right\">   0.002        </td> </tr>\n",
    "</tbody>\n",
    "</table>\n",
    "The exact value of $c_j$, $j=0,1,\\ldots,10$, appears in the first\n",
    "column while the other columns correspond to results obtained\n",
    "by three different methods:\n",
    "\n",
    "  * Column 2: The matrix and vector are converted to\n",
    "    the data structure  `mpmath.fp.matrix` and the\n",
    "    `mpmath.fp.lu_solve` function is used to solve the system.\n",
    "\n",
    "  * Column 3: The matrix and vector are converted to\n",
    "    `numpy` arrays with data type `numpy.float32`\n",
    "    (single precision floating-point number) and solved by\n",
    "    the `numpy.linalg.solve` function.\n",
    "\n",
    "  * Column 4: As column 3, but the data type is\n",
    "    `numpy.float64` (double\n",
    "    precision floating-point number).\n",
    "\n",
    "We see from the numbers in the table that\n",
    "double precision performs much better than single precision.\n",
    "Nevertheless, when plotting all these solutions the curves cannot be\n",
    "visually distinguished (!). This means that the approximations look\n",
    "perfect, despite the very wrong values of the coefficients.\n",
    "\n",
    "Increasing $N$ to 12 makes the numerical solver in `numpy`\n",
    "abort with the message: \"matrix is numerically singular\".\n",
    "A matrix has to be non-singular to be invertible, which is a requirement\n",
    "when solving a linear system. Already when the matrix is close to\n",
    "singular, it is *ill-conditioned*, which here implies that\n",
    "the numerical solution algorithms are sensitive to round-off\n",
    "errors and may produce (very) inaccurate results.\n",
    "\n",
    "The reason why the coefficient matrix is nearly singular and\n",
    "ill-conditioned is that our basis functions ${\\psi}_i(x)=x^i$ are\n",
    "nearly linearly dependent for large $i$.  That is, $x^i$ and $x^{i+1}$\n",
    "are very close for $i$ not very small. This phenomenon is\n",
    "illustrated in [Figure](#fem:approx:global:fig:illconditioning).\n",
    "There are 15 lines in this figure, but only half of them are\n",
    "visually distinguishable.\n",
    "Almost linearly dependent basis functions give rise to an\n",
    "ill-conditioned and almost singular matrix.  This fact can be\n",
    "illustrated by computing the determinant, which is indeed very close\n",
    "to zero (recall that a zero determinant implies a singular and\n",
    "non-invertible matrix): $10^{-65}$ for $N=10$ and $10^{-92}$ for\n",
    "$N=12$. Already for $N=28$ the numerical determinant computation\n",
    "returns a plain zero.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/ill_conditioning.png, width=600] The 15 first basis functions $x^i$, $i=0,\\ldots,14$. <div id=\"fem:approx:global:fig:illconditioning\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:fig:illconditioning\"></div>\n",
    "\n",
    "<p>The 15 first basis functions $x^i$, $i=0,\\ldots,14$.</p>\n",
    "<img src=\"fig/ill_conditioning.png\" width=600>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "On the other hand, the double precision `numpy` solver does run for\n",
    "$N=100$, resulting in answers that are not significantly worse than\n",
    "those in the table above, and large powers are\n",
    "associated with small coefficients (e.g., $c_j < 10^{-2}$ for $10\\leq\n",
    "j\\leq 20$ and $c_j<10^{-5}$ for $j > 20$). Even for $N=100$ the\n",
    "approximation still lies on top of the exact curve in a plot (!).\n",
    "\n",
    "The conclusion is that visual inspection of the quality of the approximation\n",
    "may not uncover fundamental numerical problems with the computations.\n",
    "However, numerical analysts have studied approximations and ill-conditioning\n",
    "for decades, and it is well known that the basis $\\{1,x,x^2,x^3,\\ldots,\\}$\n",
    "is a bad basis. The best basis from a matrix conditioning point of view\n",
    "is to have orthogonal functions such that $(\\psi_i,\\psi_j)=0$ for\n",
    "$i\\neq j$. There are many known sets of orthogonal polynomials and\n",
    "other functions.\n",
    "The functions used in the finite element method are almost orthogonal,\n",
    "and this property helps to avoid problems when solving matrix systems.\n",
    "Almost orthogonal is helpful, but not enough when it comes to\n",
    "partial differential equations, and ill-conditioning\n",
    "of the coefficient matrix is a theme when solving large-scale matrix\n",
    "systems arising from finite element discretizations.\n",
    "\n",
    "\n",
    "## Fourier series\n",
    "<div id=\"fem:approx:global:Fourier\"></div>\n",
    "\n",
    "A set of sine functions is widely used for approximating functions\n",
    "(note that the sines are orthogonal with respect to the $L_2$ inner product as can be easily\n",
    "verified using `sympy`).  Let us take"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "V = \\hbox{span}\\,\\{ \\sin \\pi x, \\sin 2\\pi x,\\ldots,\\sin (N+1)\\pi x\\}\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_i(x) = \\sin ((i+1)\\pi x),\\quad i\\in{\\mathcal{I}_s}{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An approximation to the parabola $f(x)=10(x-1)^2-1$ for $x\\in\\Omega=[1,2]$ from\n",
    "the section [Example of linear approximation](#fem:approx:global:linear) can then be computed by the\n",
    "`least_squares` function from the section [Implementation of the least squares method](#fem:approx:global:LS:code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUVf7/8deZ9EYaSWiB0EJJQg29BQOIKNhAUUQQBAW7rmV1Xdv60++uBVlXARVQiiAICnYB6TVI7y2QUFJJ78n5/XEjogYyITOZmeTzfDzmQZK5c+7nkvDm5My55yitNUIIIeyXydYFCCGEuDoJaiGEsHMS1EIIYeckqIUQws5JUAshhJ1ztkaj9evX12FhYdZoWgghaqWdO3emaq2DKnrOKkEdFhZGXFycNZoWQohaSSl1+krPydCHEELYOQlqIYSwcxLUQghh56wyRi2EqFnFxcUkJiZSUFBg61JEJdzd3WnSpAkuLi5mv0aCWohaIDExER8fH8LCwlBK2boccQVaa9LS0khMTKR58+Zmv06GPoSoBQoKCggMDJSQtnNKKQIDA6v8m48EtRC1hIS0Y7iW75P9BHVZGaz/D5zbZetKhBDCrthPUBdkQNxcWDQGcpJtXY0QwkGtWLGCN9980yJtTZs2jby8vEufDxs2jIyMDIu0XRX2E9SeATB6AeSlwxf3QkmRrSsSQthYSUlJlV8zYsQInnvuOYuc/89B/d133+Hn52eRtqvCfoIaoFEnuPl9OLMFvn/G1tUIIarolltuoWvXrkRERDBr1iwAvL29eeqpp+jSpQuxsbGkpKQAEBMTw+OPP07v3r2JjIxk+/btALz88stMnjyZIUOGcO+991JQUMB9991HVFQUnTt35pdffgHgnXfeYcKECQDs27ePyMhI8vLymDt3Lg8//DAA48ePZ8qUKQwcOJAWLVqwbt06JkyYQLt27Rg/fvyluqdMmUJ0dDQRERG89NJLAEyfPp1z584xcOBABg4cCBjLY6Smpl46f2RkJJGRkUybNg2A+Ph42rVrx6RJk4iIiGDIkCHk5+dX++/V/qbnRY2EpP2w8V1oEAXdJtq6IiEcyisrD3DwXJZF22zfqB4vDY+o9LjZs2cTEBBAfn4+3bp14/bbbyc3N5cuXbrw9ttv8+qrr/LKK6/w/vvvA5Cbm8vmzZtZv349EyZMYP/+/QDs3LmTjRs34uHhwdtvvw0YYXz48GGGDBnC0aNHefzxx4mJiWH58uW8/vrrzJw5E09Pz7/UdPHiRdasWcOKFSsYPnw4mzZt4uOPP6Zbt27s3r2bTp068frrrxMQEEBpaSmxsbHs3buXRx99lHfeeYdffvmF+vXr/6HNnTt3MmfOHLZt24bWmh49ejBgwAD8/f05duwYn3/+OR999BF33HEHX375Jffcc0+1/v7tq0f9m+tehNZDjF51/CZbVyOEMNP06dPp2LEjPXv2JCEhgWPHjmEymbjzzjsBuOeee9i4ceOl4++66y4A+vfvT1ZW1qXx3xEjRuDh4QHAxo0bGTt2LABt27alWbNmHD16FJPJxNy5cxk7diwDBgygT58+FdY0fPhwlFJERUUREhJCVFQUJpOJiIgI4uPjAfjiiy/o0qULnTt35sCBAxw8ePCq17lx40ZuvfVWvLy88Pb25rbbbmPDhg0ANG/enE6dOgHQtWvXS+eoDvvrUQOYnOD2j+GjWPhiLExeC35NbV2VEA7BnJ6vNaxdu5ZVq1axZcsWPD09iYmJqXC+8OXT0/48Ve23z728vC597WobcB87dgxvb2/OnTt3xWPc3NwAMJlMlz7+7fOSkhJOnTrFW2+9xY4dO/D392f8+PGVznO+Wk2Xn8PJyckiQx/22aMGcPeFuz6H0hJYdDcU5dq6IiHEVWRmZuLv74+npyeHDx9m69atAJSVlbF06VIAFi5cSN++fS+9ZvHixYDRQ/X19cXX1/cv7fbv358FCxYAcPToUc6cOUObNm3IzMzkscceY/369aSlpV06R1VlZWXh5eWFr68vSUlJfP/995ee8/HxITs7u8KavvrqK/Ly8sjNzWX58uX069fvms5vDvvsUf+mfmsY+QksGAVfPwQj54BM6hfCLg0dOpQZM2bQoUMH2rRpQ8+ePQGjd3zgwAG6du2Kr6/vpXAG8Pf3p3fv3mRlZTF79uwK2506dSoPPvggUVFRODs7M3fuXNzc3JgyZQpTp04lPDycTz75hIEDB9K/f/8q192xY0c6d+5MREQELVq0+MMQyuTJk7nhhhto2LDhpTcxAbp06cL48ePp3r07APfffz+dO3e2yDBHRdTVuvDXKjo6Wlt044CN02DVSxD7T+j3lOXaFaKWOHToEO3atbN1GRXy9vYmJyfnL1+PiYnhrbfeIjo62gZV2VZF3y+l1E6tdYV/GfY79HG5Po9B1ChY/Roc+cHW1QghRI2y76GP3ygFI/4LqUfhy/th0moIamPrqoQQZqioNw3Gm4/CPI7RowZw8YDRC8HFHT4fbdzBKIQQdYDjBDWAbxO4cwFkJspt5kKIOsOxghqgaQ+4+X8QvwG++xtY4c1QIYSwJ44xRv1nHe6AlCOw4S1jrLrXQ7auSAghrMbxetS/GfgCtBsBP74AR3+0dTVCCCtZu3YtmzdvtnUZNmVWUCulnlBKHVBK7VdKfa6Ucrd2YZUymeDWGdCwAyydAElXvzdfCOGYJKjNCGqlVGPgUSBaax0JOAGjrV2YWVy94K5F4OoNC++EnBRbVyREnTV//ny6d+9Op06deOCBBzh9+jStW7cmNTWVsrIy+vXrx08//QRUvBwqwA8//ECXLl3o2LEjsbGxxMfHM2PGDN599106dep0aeGjusbcMWpnwEMpVQx4AldeAaWm1WtkrAkyZxgsHgPjVoKzW+WvE6K2+v45uLDPsm02iIIbrrxryqFDh1i8eDGbNm3CxcWFqVOnsm7dOp599lkefPBBevToQfv27RkyZAhQ8XKoZWVlTJo0ifXr19O8eXPS09MJCAjgwQcfxNvbm7/97W+WvSYHUmlQa63PKqXeAs4A+cBPWuuf/nycUmoyMBmgadMaXumucRe49UNYMh5WPGoMiciaIELUmNWrV7Nz5066desGQH5+PsHBwbz88sssWbKEGTNmsHv37kvHT58+neXLlwNcWg41JSWF/v3707x5cwACAgJq/kLsVKVBrZTyB24GmgMZwBKl1D1a6/mXH6e1ngXMAmOtDyvUenURt0LqMfjldQgKlzVBRN11lZ6vtWitGTduHG+88cYfvp6Xl0diYiJg3KHo4+NzxeVQtdayk/oVmPNm4iDglNY6RWtdDCwDelu3rGvU/+nyNUFehYMrbF2NEHVGbGwsS5cuJTnZ2Jg6PT2d06dP8+yzzzJmzBheffVVJk2aBFx5OdRevXqxbt06Tp06dakNuPJSo3WJOUF9BuiplPJUxn93scAh65Z1jZSCEe9Dk26wbDKc3WnrioSoE9q3b8+//vUvhgwZQocOHRg8eDDx8fHs2LHjUli7uroyZ84chg4dSklJCR06dODFF1+8tBxqUFAQs2bN4rbbbqNjx46XdoUZPnw4y5cvr9NvJpq1zKlS6hXgTqAE2AXcr7UuvNLxFl/mtKpykuHjQVCcB/evAv8w29UiRA2w52VOxV9ZZZlTrfVLWuu2WutIrfXYq4W0XfAOhjFLoLQIFtwB+RdtXZEQQlwzx70zsTJBbYwFnNJPwuKxsoCTEMJh1d6gBmje7/cFnFY+Kgs4iVrNGrs1Ccu7lu9T7Q5qgI53QszzsOdzWPd/tq5GCKtwd3cnLS1NwtrOaa1JS0vD3b1qq3A45up5VTXgGbgYD2vfAL9m0OkuW1ckhEU1adKExMREUlJkGQV75+7uTpMmTar0mroR1ErB8PcgKxFWPAK+jaF51XcrFsJeubi4XLqjT9Q+tX/o4zfOrnDHPAhsCYvugeTDtq5ICCHMUneCGsDDD+7+wli0acEoyE6ydUVCCFGpuhXUAP7N4O5FkJsCn98JhRXvkCyEEPai7gU1QOOuMHI2nN9jrLhXWmzrioQQ4orqZlADtB0GN74Nx3+GlY/JHGshhN2qG7M+riR6AmRfMOZX+zSE2BdtXZEQQvxF3Q5qgJi/Q/Z5Y0fzeg2h2/0WbT6vqISjSTlczCsCDcH13GgZ5I27i5NFzyOEqL0kqJWCG981Vtz79m/gHQLthlerybyiEpbEJbJyzzl2nrn4l1EVVycTnZv6cWvnxgzv2AgvN/k2CCGuzKxlTqvK5sucXouiXPh0hLHX3L1fQ7NeVW6itEwzb0s8760+xsW8Yto28GFw+xAiGvkSXM/Yx/FCZgF7EjJYfTiZ48k5+Hu6MDWmFeN6h+HqXHffMhCirrvaMqcS1JfLTYPZQ4ypexN+guC2Zr80IT2PRz7fxe6EDPq2qs8Tg1vTtdmV93zTWrPz9EXeW32MDcdSaRPiw39GdaBDEz9LXIkQwsFIUFfFxXj4ZAiYXGDiT8bt5pXYfDyVqQt/pbRM869bIhnRsVGV9n5bfSiJF5bvJz23iNduieDObjW8ObAQwuaqvXFAneIfBmOWQkEmLBgJ+RlXPfyH/RcYP2cHwT5urHy4Lzd3alzlDTpj24Xw/WP96NEigGe/3MeLX+2ntEymCwohDBLUFWnYAUbPN3Y1XzQGigsqPGzVwSQeWvgrEY3r8cUDvQir73XNp/T3cmXO+G5M7t+CeVtP8+QXuykpLbvm9oQQtYcE9ZW0iIFbZ8DpjbB0ApSW/OHpuPh0Hlr4K5GN6jFvYg/8PF2rfUpnJxPPD2vH09e34evd53h00S6KJayFqPMkqK8maiTc8B848q2xQ0yZEZoXMgt4cP5OGvt5MOe+7nhbeHrdQwNb8Y8b2/Hdvgs8v2yfLAYvRB0nE3gr02My5Kcbmw54+FN03atMXbCT/KJSFk3uSYBX9XvSFbm/XwuyCkqYvvoYjfw8eGJwuFXOI4SwfxLU5hjwLOSlw5b32XC6mF/PDOB/d3ehVbCPVU/7xKDWnMvI573VxwgN8GRk16rtCiGEqB0kqM2hFAx9k+TkJGLjZ/J+eD1u7HBjDZxW8cZtUZzLyOeF5fto28CHyMa+Vj+vEMK+yBi1mXKKyxh5fgxbnLpx45m3YP+XNXJeFycT0+/qTICXK1MW7CQzT5ZkFaKukaA205vfHyIhqxi3uz9DNesNyx6A46tq5Nz1vd34YEwXLmQW8OQXu+XNRSHqGAlqM+w8nc78rWeY0Kc5XVo2grs+h+B2sHgsnNlWIzV0burP88PasfpwMvO3namRcwoh7IMEdSXKyjSvrDxISD03nvxt5oW7L9yzzFjDeuEoSDpQI7WM6xVG//AgXv/2ICdSZAsxIeoKCepKfPlrInsTM3nuhrZ/XI7UOwju/QpcvGDerZB2wuq1mEyK/4zsgIeLE48v2i03wwhRR0hQX0VOYQn//vEInUL9uLljBYsz+TWFscuNPRc/uxkyEqxeU0g9d/7frVHsO5vJRxtOWv18QgjbMyuolVJ+SqmlSqnDSqlDSqmqL9bsgGatP0lKdiEvDW+PyXSFhZaC2xo964Is+GyEsbWXld0Q1ZAbIhvw3qpjnErNtfr5hBC2ZW6P+j3gB611W6AjcMh6JdmHi7lFzN54ihsiG9C5qf/VD27YEe750tgl5rObITfV6vW9MiICV2cTf1+2V2aBCFHLVRrUSql6QH/gEwCtdZHW+uprf9YCszacJLeohMcHmXnrdmg3uHuxsZ71vFsg/6JV6wuu587fb2jH1pPpfBFn/SEXIYTtmNOjbgGkAHOUUruUUh8rpf6ynqdSarJSKk4pFZeSkmLxQmtSak4hn26O56YOjWjToAq3iYf1hdELIOUIzB8JhdnWKxIY3S2U7s0DeP3bQ6RkF1r1XEII2zEnqJ2BLsCHWuvOQC7w3J8P0lrP0lpHa62jg4KCLFxmzZq1/iQFxaU8Pqh11V/cahCMmgvndsHCO6Eoz+L1/cZkUvy/W6PIKyrlPz8ettp5hBC2ZU5QJwKJWuvf7uxYihHctVJmXjHzt55mRMdGtAzyvrZG2t4It82C05th8RgosV5vt1WwNxP6NueLuER2J9T6ESkh6qRKg1prfQFIUEq1Kf9SLHDQqlXZ0Pxtp8krKuWBAS2r11DUSLj5fTixBpaMN6bwWckj17UiyMeNl1ccoEy28BKi1jF31scjwAKl1F6gE/D/rFeS7RQUlzJnUzz9w4No17Be9RvsfA8MewuOfAfLJkFZafXbrICPuwvPDW3L7oQMlu06a5VzCCFsx6xlTrXWu4EKd8etTZbvOktqTiEP9m9huUa7T4LifPj5RXByhVs+BJOT5dovd2vnxszfdpo3vz/M9REh+Li7WPwcQgjbkDsTy5WVaT5af5Koxr70ahlo2cb7PArXvQh7F8NXU63SszaZFC8PjyA1p5CZ6+SORSFqEwnqcuuPpXAyNZf7+zVHqSvchVgd/f8GA/8BexfB1w9ZJaw7hvoxomMjPt54kguZFe+cLoRwPBLU5eZtOU19bzduiGxovZMMeBoGvgB7PoevH7ZKWD99fRtKyzTv/nzU4m0LIWxDghpISM9jzZFk7u4eiquzlf9KBjwDMc/DnoWw4pFLO5tbSmiAJ/f2CmPJzgSOJln3hhshRM2QoAbmbz2NSSnu7tGsZk4Y8yzE/B12L7BKWD88sBVebs783/dyE4wQtUGdD+qC4lIWxyVwfUQIDXzda+7EMc/BgOdg93xYadmw9vdyZWpMK1YfTmbLiTSLtSuEsI06H9Tf7TtPRl4xY3uG1fzJY56D/s/Arvmw8lGLhvV9fcJo6OvOG98fktX1hHBwdT6ov4hLICzQk54tAmr+5ErBwOeh/9Owax5885jFwtrdxYknBoezNzGTHw8kWaRNIYRt1OmgPp2Wy9aT6YyKDrXOlDxzKGXMBOn3N/j1s/KetWVmg9zWuTEtgrx45+cjlMqt5UI4rDod1Et3JmJScFuXCrbZqklKwXX/KB8GmQdfTYHSkmo36+xk4olB4RxNyuGbvecsUKgQwhbqbFCXlmmW7kykf3gQDX09bF1OeVi/YAT23sXw5USLLOR0Y1RD2jbwYdqqY5TIZrhCOKQ6G9Qbj6dyPrOAUV1DbV3KH/V/Gob8Cw5+BV+Mq/YSqSaT4snB4ZxKzZUFm4RwUHU2qL+IS8DP04VB7YNtXcpf9X6kfNW9b2HRGGNRp2oY3D6EDk18eW/VMQpLrLOCnxDCeupkUGfkFfHzgSRu6dQYN2fLr2RnEd0nwfD34Piq8p1irn23caUUTw1pw9mMfL7YIfsrCuFo6mRQr9x7nqLSMkZFN7F1KVfXdbyxLGr8hmrvwdi/dX26hfnz3zXHKSiWXrUQjqROBvWK3WdpHexNe0tsDmBtne6C2z+GhG0w71bIv7bttn7rVSdnFzJ/62kLFymEsKY6F9RnM/LZEX+RER0b2W7udFVF3g53fArndsOnwyE39Zqa6dkikL6t6vPh2hPkFVV/+p8QombUuaD+Zo8xn3hEp0Y2rqSK2g2H0Qsh9SjMuQEyE6+pmScGtyYtt4jPtkivWghHUeeCesWec3QM9aNZoJetS6m68CFwzzLIOg+zh0LaiSo30bVZAAPCg5i57gQ5hdKrFsIR1KmgPp6cw4FzWYzo6GC96cuF9YHxK6E4D2ZfDxf2VbmJJwaHczGvmE83x1u+PiGExdWpoF6x5xxKwU0drLiLS01o1Bnu+8HYLHfujXBmW5Ve3inUj9i2wcxaf5Ksgurf/SiEsK46E9Raa1buOUevFoGE1KvBdaetJSgcJvwAnoEw7xY4vrpKL39icDiZ+cXM2RhvnfqEEBZTZ4J6/9ksTqXmOvawx5/5NYUJP0JAS+OmmANfmf3SyMa+DGkfwscbT5KZL71qIexZnQnqb/edx9mkGBrZwNalWJZ3MIz/Bhp3gaX3wa/zzH7p44PCyS4o4ZONp6xYoBCiuupEUGut+WH/eXq1DMTP09XW5Viehx+MXQ4tYmDFw7BxGpixq0v7RvUYFtWA2RtPkZFXZPUyhRDXpk4E9eEL2cSn5dW+3vTlXL3grkUQcRusegl+fN6s3WIeiw0nt6iEjzacrIEihRDXok4E9Q/7L6AUDGlfi4MawNkNbv8EekyBrR8Ya1pXskxqmwY+3BjVkDmb4knPlV61EPaozgR1t7AAgnzcbF2K9ZlMMPQNGPQKHFgGC0ZBQdZVX/L4oNbkF5cyc33Vb6ARQlhfrQ/qkyk5HEnKZmhELe9NX04p6Ps43DID4jfC3GGQfeUNblsF+3Bzx0Z8tvk0KdnV26hACGF5Zge1UspJKbVLKfWNNQuytO/3XwCo3ePTV9LpLrh7sXGr+SeDr3rL+aOxrSksKWXmOulVC2FvqtKjfgw4ZK1CrOXHAxfoGOpHIz872BfRFloPhnHfQFGOEdZnd1Z4WIsgb27t3IR5W0+TnFVQw0UKIa7GrKBWSjUBbgQ+tm45lpV4MY+9iZncUBd705dr0hUm/GTMDJk7HI6tqvCwR2NbUVKm+WCt9KqFsCfm9qinAc8AV5zvpZSarJSKU0rFpaSkWKS46vrpgDEue31dGp++kvqtYOIqCGwBC++AuDl/OaRZoBcjuzRh4fYznM+s3j6NQgjLqTSolVI3Acla64p/Zy6ntZ6ltY7WWkcHBQVZrMDqWHUoidbB3jSv74BLmlqDTwjc9z20vA6+eRx+fukvc60fvq4VZWWaD36RXrUQ9sKcHnUfYIRSKh5YBFynlJpv1aosIKugmO2n0oltF2LrUuyLm49xY0z0BNg0Db6cAMW/j0mHBnhyR7dQFu04w9kM6VULYQ8qDWqt9d+11k201mHAaGCN1voeq1dWTeuOpFBSpoltF2zrUuyPkzPc+A4MfhUOLIfPRkBu2qWnHxrYCoXi/TXHbVikEOI3tXYe9epDSfh7utClqb+tS7FPSkGfx2DUXGMvxk8GXZq+19jPg9HdQ1kSl0BCep5t6xRCVC2otdZrtdY3WasYSykpLeOXIykMbBOMk8lBNrC1lYhbYdxKKMiEjwfBma0ATI1phcmk+O+aYzYuUAhRK3vUO09fJDO/WManzdW0B0z8GTz84dMRsP9LGvi6M6ZHU7789Szxqbm2rlCIOq1WBvXqw8m4OCn6h9e3dSmOI7Al3L+qfF3rCbD2TaYMaIGLk+K/MlYthE3VyqBedSiJHs0D8XF3sXUpjsUzAO79GjreBWvfIPjHKdzXLYTluxI5mZJj6+qEqLNqXVCfSs3lZEquzPa4Vs5ucMuH5avvfcWTZ58g1DmT6atlrFoIW6l1Qb36kHE34iAZn752v62+N3ohLheP8437i5zau4Hjydm2rkyIOqnWBfWqQ0mEh3gTGuBp61IcX9thMPEnPD09WezyKuuXz7R1RULUSbUqqDPzi9kRf1Fme1hSSAROk38htV57Jpx/jdSVf73tXAhhXbUqqDccS6G0TBPbVsanLcqrPt6TvmW5jqH+zmmwdDwUyZQ9IWpKrQrqdUdSqOfuTKdQP1uXUuv41fMhvs+/ea14DPrgCvhkCKSfsnVZQtQJtSaotdasO5pCv/AgnJ1qzWXZlQn9WrDE5Wbea/AGZCbCrBg4vtrWZQlR69WaRDt0Ppvk7EIGhNvHEqu1ka+HC5P6tWBafFMODV8B9RrDgpGw8V3Q2tblCVFr1ZqgXnfU2KxAgtq6xvcJI9DLlZc35qEn/gTtb4FVL8OS8VAoN8UIYQ21KKiTadvAh5B67rYupVbzcXfh8cHhbDuVzs/Hc2DkbGO51EMrKt1AVwhxbWpFUGcXFBMXf5GYNjLboybc1S2UVsHevPn9YYrLtLFc6j1fQvZ5+GggHPvZ1iUKUavUiqDefCKNkjItwx41xNnJxPPD2nIyNZeF284YX2x5HUxeC75NYcEoWPcfmW8thIXUiqBedzQFL1cnujaTTQJqysA2wfRuGci0VUfJzC82vugfBhN/gqhR8Mu/jE1089JtWqcQtYHDB7XWmnVHUujTqj6uzg5/OQ5DKcULN7YjI7+YD365bBlUV0+4bZax1depdTCzPyTG2a5QIWoBh0+2Eyk5nM3IZ0AbGfaoaRGNfLm9SxPmbIrndNpldyoqBd0mwoQfjY9nD4VtM2UKnxDXyOGDeu0RY1pe/9YS1Lbw9PVtcHFSvLziAPrPQdy4CzywHloNgu+fMabwFWTZpE4hHJnDB/W6oym0DPKS1fJsJKSeO08MDueXIymsOpT81wM8/GH0QmN960MrjVkhSQdqvlAhHJhDB3V+USnbTqUzIFym5dnSuN5hhId488rKAxQUl/71AJPJWN963ErjppiPYmHXgpovVAgH5dBBvSM+naKSMvrJ3og25eJk4pURkSRezOeDtVe54SWsDzy4AZpEw9dTYdlkKJTNCISojEMH9cbjqbg6mejRPMDWpdR5vVoGcnOnRsxYd+Lqu5Z7Bxv7MsY8D/uWGLNCzv5ac4UK4YAcOqg3HEulSzM/PF2dbV2KAJ4f1g4Xk+LllRW8sXg5kxPEPAvjv4WSImPJ1M3/lRtkhLgChw3q1JxCDp3Pop/M9rAbIfXceXJIG9YeSWHFnnOVv6BZb2MoJPx6+OkfsHAU5FTwhqQQdZzDBvWm46kA9Gkl49P2ZHzvMDqF+vHyigOk5hRW/gLPALhzvnGDTPxG+LCPrHEtxJ84bFBvPJaKr4cLUY19bV2KuIyTSfHvkR3IKSzh5RVmTsP77QaZSb8YwT3/Nvj5n8awiBDCMYNaa82m46n0bhmIk0nZuhzxJ+EhPjxyXWu+2Xuenw5cMP+FIe2NsO56H2x6Dz4ZBMmHrVeoEA7CIYP6ZGou5zIL6Ntahj3s1ZSYlrRt4MMLX+0nPbcKPWNXTxg+zRgOyUyEWQNg6wx5o1HUaZUGtVIqVCn1i1LqkFLqgFLqsZoo7Go2HjPGp/vK+LTdcnEy8fYdHcnIK+Lvy/ZefRZIRdoNhylboPkA+OFZmHcLZJ61TrFC2DlzetQlwFNa63ZAT+AhpVR765Z1dRuPpxIa4EGzQC9bliEqEdHIl2eub8uPB5JYvCOh6gca6YMAABgySURBVA34hMDdi+GmaZC4Az7sBfuWWr5QIexcpUGttT6vtf61/ONs4BDQ2NqFXUlJaRlbT6TRt5VMy3MEE/s2p0+rQF5ZeZCTKdewp6JSEH0fPLgR6ofDlxNh6QRZ51rUKVUao1ZKhQGdgW0VPDdZKRWnlIpLSUmxTHUV2JOYQXZhCf1kfNohmEyKt0d1ws3FxGOLdlNYUsFaIOYIbAn3/QDX/QMOfi3T+ESdYnZQK6W8gS+Bx7XWf1mrUms9S2sdrbWODgqyXm9347E0lIJeLQKtdg5hWQ183fn37R3YdzaTl1ccvPaGnJyh/9Nw/ypw8zam8a14BAoyLVesEHbIrKBWSrlghPQCrfUy65Z0dRuPpxDV2Bd/L1dbliGqaEhEA6bGtOTz7WdYvONM9Rpr1NlY57rPY7BrPnzQSzbUFbWaObM+FPAJcEhr/Y71S7qynMISdp3JkNkeDuqpIW3o17o+L359gD0JGdVrzMUDBr8KE1eBmw8sGAnLp0D+RcsUK4QdMadH3QcYC1ynlNpd/hhm5boqtO2ksdu4zJ92TE4mxXujOxPk7caU+TtJziqofqNNuhq9635Pwd7F8L+ecPi76rcrhB0xZ9bHRq210lp30Fp3Kn/Y5F/ChmOpuLuYZLdxBxbg5crMsV3JyC9m/JwdZBcUV79RZzeI/SdMWgNe9WHRXfDl/TIzRNQaDnVn4pYTaXQLC8DN2cnWpYhqiGzsywdjunAkKZupC36lqMRCdx026mTcgh7zdziwHP7X3Zh3LZvqCgfnMEGdmlPIkaRserWU2R61QUybYN68LYoNx1J57su9lJVZKEydXSHmOZi8DnybGPOu598O6acs074QNuAwQb31ZBoAvVvK+HRtMSo6lKcGh7Ns11n+8fV+y4U1QINIuH813PBvSNgOH/SEDe9AqQWGWoSoYQ4T1JtPpOHj5kxko3q2LkVY0MPXtWJKTEsWbjvDi5YOa5MT9HgAHtoGrQfD6leMrb/O/OV+LSHsmsME9ZYTafRoEYCzk8OULMyglOKZ69swJaYlC7ad4W9L9lBcauGV8nwbG6vxjf4cCrJg9hD45gnIr+YUQSFqiEOk3rmMfE6l5tJT7kaslX4L6yfLh0EmzN1BTmGJ5U/UdpjRu+71MOycC+93g71L5M1GYfccIqi3nJDx6dpOKcWjsa35v9uj2HwijZEfbr76bubXys0brn/dmB3i2wSW3Q9zhsGF/ZY/lxAW4hhBfTINf08X2jbwsXUpwsru7NaUOeO7cSGrgOHvb2T1oSTrnKhRJ+PNxuHvQcphmNkPvntG7mwUdsnug1przZYTafRqGYhJtt2qE/qHB7Hy4b6E+nsy8dM4Xli+j1xrDIWYTNB1PDyyE6Inwo6P4L9d4dfPZEcZYVfsPqjPpOdxNiOfXjLsUaeEBniybGpvJvdvwcLtZxj63npWH0qq+k4x5vAMgBvfMuZeB7Y2VuT7ZBCc3Wn5cwlxDew+qDeXj0/LsqZ1j7uLE88Pa8fiyb1wcTIx8dM4xs/ZwaHzf1ll1zIadoAJP8Cts4z9Gj+6DpY/WOUtwPKLSklIz+NYUjb7z2ay83Q6exIyOHwhi1OpuWTmF1vnPxxRaylr/MBER0fruLg4i7T1yOe72HYyjW3Px2Is5CfqoqKSMj7bEs97q4+RXVDCoHbBTO7fkm5h/tb5uSjIgg1vwdYPQTlB70eMZVXdvNFacz6zgOPJORxPzuFEivG4kFlASnYhuUWVb47g4eJESD03WgZ507ahD20b1KNLM38a+3lY/lqEQ1BK7dRaR1f4nD0Htdaabq+vpm+rQKaN7myByoSjy8wr5tMt8czedIqMvGJa1Pfi9q5NuD4ihJZB3hYN7eLSMs6eOozb2tdomPgdWU4BzPUYy0dZPcku+v3fja+HC62CvWnk50F9b1fqe7tR39sVT1dn3F2ccHU2UVpWRmFxGfnFpaTlFJGUVcD5rAKOJxkhX1J+o0/z+l70aRVIbLsQ+raqj4vcN1BnOGxQH0vKZvC76/n37R24o1uoBSoTtUVeUQnf7j3PkrhEtscbq+SFBnjQLSyAzqF+hIf4EBrgSUg9d5yu8Ca01pqsghJSsgtIzirkdHoep1JzOZmSw8nUXM6k5V0K0C7qKK+4LyRKH+WCR2v2RT6Nd9tBtAr2pr63a7X+gygsKeVYUg7bTqWz6Xgq206mkVtUSoCXKzdENuCO6FA6hvpdc/vCMThsUH+6OZ6XVhxgwzMDCQ3wtEBlojY6l5HPmsPJrD2Swu6Ei6TmFF16zsVJ4efpioeLEx4uxqqLBSWlFBSXkplfTEHxH2d3uDqbaB7oRfP6XjQP8qJVkDetgr1pGeyNt6sTHFgGq16GjDPQegjEvmSsK2JBhSWlrD+ayoo951h1MIn84lI6N/Xjvj7NuSGygfSyaymHDeoH5sVx4FwWG5+9zgJVibpAa83ZjHxOpuSSeDGfhIt5ZOQVU1BcSl5RCQqFm4sJd2cn6nk4E+zjTnA9N4J83Gga4EkjX4/Kp4EWF8C2GbDxHWMsO2oUDHweAppb/HpyCktYGpfA3M3xxKflERrgwWOx4dzaufEVf1MQjskhg7qsTNP5tZ+5PiKEf4/saKHKhLCg/Iuw6T3YOgPKiqHrfcbmuz4hFj9VWZlmzeFkpq0+yv6zWbQM8uLp69tyfUSIvMleS1wtqO32d6iD57PIzC+W28aF/fLwh0Evw6O7oMu9sHMOTO8Eq1+z+M7oJpNiUPsQVj7clxn3dMGkFA/O38m4OTs4mZJj0XMJ+2O3Qf3b+h6yUYCwe/Uawk3vwkPboc0wY1rfex1hw9tQmG3RUymlGBrZkO8f68c/b2rPrtMXGTptA+/8dMRyO+UIu2O3Qb35RCotgrwIqedu61KEME9gSxj5ibHZbpPusPpVmBZllcB2djIxoW9zVv9tADd2aMj0NccZ8f5GDpyzbE9e2Ae7DOri0jK2n0qnt/SmhSNq2BHGfGFstnt5YK9/y3jz0YKCfdx5985OfHxvNGm5Rdz8/ibeX3OMUktuwCBszi6Dem9iJrlFpTI+LRxb465/DOw1r8F7HawS2IPah/DzE/25Iaohb/10lHGzt5OSXWjRcwjbscug/m1/RNkoQNQKlwd2aA8jsKdFwpp/QW6qxU7j5+nK9NGd+L/bo9gRn86w6RvYfMJy7Qvbscug3nwilXYN6xHg5WrrUoSwnMZd4e7FxqYFYf2MnvW7kfDd03DxtEVOoZTizm5N+frhPvi4O3PPx9v43y/HZREoB2d3QV1QXEpc/EVZLU/UXo27wOgFxiyRyNshbg5M7wzLJkPSAYucom2Deqx8uC83dmjEf348wqOLdpNvxmJRwj7ZXVDvOpNBYUmZvJEoar+gcLjlf/DYHug5BQ59Ax/2hoV3QvzGau/l6OXmzPTRnXhmaBu+2XuOO2Zu4XxmvoWKFzXJ7oJ6y4lUTAq6twiwdSlC1AzfxsY+jk/sh4EvQMJ2mHsjzOwPuxdCybW/KaiUYmpMKz4aG83JlBxGvL+JPQmy+7qjsbug3nwijagmftRzd7F1KULULM8AGPAMPHHA2MuxtAi+mmKMY699E3KSr7npQe1DWP5QH9ycTYyetZU1h620F6WwCrsK6ryiEnYnZMiwh6jbXD2NvRynboWxy42NeNe+Ae9GwFdT4fzea2o2PMSHZVN70yrYm0mf7WTR9jOWrVtYjVlBrZQaqpQ6opQ6rpR6zlrF7Ii/SEmZljcShQBQClpeB2OWwMNxxnoiB5YbO6Z/MgT2LILiqo05B/u4s2hyT/q2qs9zy/bxzs9HZUaIA6g0qJVSTsD/gBuA9sBdSqn21ihm84lUXJwU0WH+1mheCMdVvzXc+DY8eRCGvA55abD8AXi7LfzwPKQeM7spLzdnPh4XzaiuTZi++hjPLN1LcamsE2LPzOlRdweOa61Paq2LgEXAzdYoZsuJNDqH+uPp6myN5oVwfB7+0Ptho4c9biW0HAjbZ8L70TD3Jti/DEqKKm3GxcnEv0d24NHY1izZmcj9n8aRV1RSAxdQe/165iIfbzhplbbNCerGQMJlnyeWf+0PlFKTlVJxSqm4lJSUKhdSUFzKmfQ8WS1PCHMoBc37w6i58MRBiP0nZJyGpffBO23h++cqHctWSvHk4HDeuC2KDcdSuPujbVzMrTzkxV8dOJfJ+Nnbmbf1NDmFlv8Pr9KNA5RSo4Drtdb3l38+FuiutX7kSq+51o0DSkrLKCwpw8tNetRCVFlZKZxYA7vmwZHvjVkjIVHQ6W5jFxrvoCu+9McDF3jk812E+nvw2cQesht6FRxPzubOmVtxczbxxYO9aOJ/bdsGVnfjgETg8p1lmwDnrqmSSjg7mSSkhbhWJidoPRju+AyeOgLD3gInF/jx70Yv+/O74dDKCudlXx/RgHkTupOcXcjtH2zmaJJll2Wtrc6k5THm420opVgwqec1h3RlzOlROwNHgVjgLLADuFtrfcV7XS21Z6IQwgKSDxk3zuxdDDlJ4O4L7YZDxG3QfAA4/d45OnQ+i3tnb6ewuJTZ47sRHSY3nl3J+cx87pi5heyCEhZN7knbBvWq1V6190xUSg0DpgFOwGyt9etXO16CWgg7VFoCJ9fC/qXG7epF2eBZH9rfbKw50rQXmEwkpOdx7+ztnMvI54MxXYhtZ/k9IB1dak4hd8zcQnJWIQsn9aBDE79qt+mQm9sKIayouACO/wz7v4QjP0BJPvg0gohboO1NpAZ04r5Pd3HwfBZv3hbFqOjQytusIzLzihn90VZOpebw2YQedG9umd86rhbUMiAsRF3k4m4Mf7QbDoU5cPQHY2rfjo9h6wfU9wxkWcsh/Jc2vLi0gNScIh4c0KLO73iemV/MvXO2cyI5h4/HRVsspCsjPWohxO8Ks+H4ajj8LRz9EQozKVJu/FISRUHLoQy/fTwmnyvPHqnNMvOLuXf2dg6ey+SDMV0Z3N6yQ0Iy9CGEqLqSIji9CX34W7J3f0W94hTKUNCoM6ZWg6DVIGMzBKfa/4t5VkExYz+xXkiDBLUQopp0WRlLvvmGhG1fM9zrIK2LD6N0mTGDpEWMEdotY40lW2uZmghpkDFqIUQ1KZOJO0aMYFGDTgxdvo+ejZyY2Scbn4R1xlDJwa+NAwNaQlhfY6uxsL5Qr6FtC68maw93mEt61EKIKvmp/C7Gxv4efDahO038PIy52ifWGDvTnN4MhZnGwZcHd7PeDtXjTsku5N7Z2zmenF0jIS1DH0IIi9p+Kp2Jn+7Ay9WZTyd0p00Dn9+fLCuFC/uM0P5zcPs0gibR0KSb8WjUCVzs73b1hPQ8xn6yjaSsQmbd25V+ra3/BqoEtRDC4g5fyOLeT7ZTUFzKx+O6XXmqWlkpXNhrbDGWuMN4XIw3njM5Q0jk76HdIAqC2oKzW41dx58dS8rmnk+2kV9Uypz7utO1Wc0suyxBLYSwioT0PMbN3k7CxTxevTmSu7o3Ne+FOSlwNu734D77KxTlGM+ZnKF+GyO0f3uERIKX9VfW3H4qncnz4nBxMjFvYvdq3xZeFRLUQgirycwr5pFFu1h/NIUxPZry0vAIXJ2ruMtfWRlcPGX0vC/s+/2Rff73YzwDoX64sYlC/fDfP/ZrZixIVU3Lfk3k2S/3EurvyZz7utEs0KvabVaFBLUQwqpKyzT/+fEIM9adILqZPx+M6UJwPffqN5yTAkn7IOmAsYtN6jFIPQp5qb8f4+QK/mHg1/RPj2bGn15BxvrdV1BWppm26ijT1xynV4tAZtzTFV/Pmt9cW4JaCFEjVuw5xzNL9+Dl6sxbozoysG2wdU6Ul/57aKceMca8M84Yj/yLfzzW2R28Q8ofwX/4OMclkGlb0vnldBExHVry7C09cXX3vGqwW4sEtRCixhxLyuaRz3dx+EI29/UJ47kb2uLmXP2hCbMVZkNGwu/BnXEacpKNJV5/+zM//cqvd3I1buT57eHiaby56ez+pz89jPW+TU6gTMbD1Rv6PHpNZUtQCyFqVEFxKW9+f5i5m+Np17Aeb4/qSPtGNffG3NWUlJbx0dojLFwTR7hXPi/EBNHCpxQKMit4ZBgrDZYUGBsu/PnP0kLQZb8/vILhafM3Gr6cBLUQwibWHE7imaV7uZhXzKR+LXgstjUerjXYu/6T48k5PLVkD3sSMhgW1YDXb4nC38vVcifQ+pqHTSSohRA2k5FXxBvfHWZxXAKhAR68MKwd10c0qNElU3MLS/hw7QlmbTiJp6sTr90cyU0dGtrVsq0S1EIIm9t6Mo0Xv9rPseQcOoX68dwNbenZwrpzo0vLNMt+TeQ/Px4hObuQmzs14oUb2xHsY4EZKRYmQS2EsAslpWUs+/Us7/x8lAtZBXQL82dCn+YMbh+Cs1MV515fRUFxKV/tOsuMdSeIT8ujY6gf/7ypfY3dZXgtJKiFEHaloLiUBdvOMHfzKRLS82ns58HtXRpzU8dGhIf4VN5ABbTW7D+bxZe/JrJ811ky84uJauzL1JiWXB/RAJPJfoY5KiJBLYSwS6VlmlWHkvhsSzxbTqRRpiEs0JNeLQPpFhZAeIgPLYO8K3wDMrugmNNpeew7m8muMxdZdzSFpKxCXJ1MDIkI4a7uTendMtCuxqGvRoJaCGH3krML+H7fBdYfTWH7qXSyC0suPeft5oyfpwtOJkVJqSYrv/gPz/t6uNCnVSCxbUO4rm2wZWdy1BDZOEAIYfeCfdwZ1zuMcb3DKC3THE/O4XhyDidTckjPKyIjr5gyrXE2mfBxd6ahrzuN/T2IbORLs0BPh+k5XwsJaiGE3XEyKdo08PnjOtd1mOXeZhVCCGEVEtRCCGHnJKiFEMLOSVALIYSdk6AWQgg7J0EthBB2ToJaCCHsnAS1EELYOavcQq6USgFOX+PL6wOplR5Vu8g113517XpBrrmqmmmtgyp6wipBXR1Kqbgr3e9eW8k113517XpBrtmSZOhDCCHsnAS1EELYOXsM6lm2LsAG5Jprv7p2vSDXbDF2N0YthBDij+yxRy2EEOIyEtRCCGHnbBbUSqmhSqkjSqnjSqnnKnjeTSm1uPz5bUqpsJqv0nLMuN4nlVIHlVJ7lVKrlVLNbFGnJVV2zZcdN1IppZVSDj+Vy5xrVkrdUf69PqCUWljTNVqaGT/bTZVSvyildpX/fA+zRZ2WopSarZRKVkrtv8LzSik1vfzvY69Sqku1T6q1rvEH4AScAFoArsAeoP2fjpkKzCj/eDSw2Ba11uD1DgQ8yz+e4sjXa+41lx/nA6wHtgLRtq67Br7PrYFdgH/558G2rrsGrnkWMKX84/ZAvK3rruY19we6APuv8Pww4HtAAT2BbdU9p6161N2B41rrk1rrImARcPOfjrkZ+LT846VArHLcTdEqvV6t9S9a67zyT7cCTWq4Rksz53sM8Brwb6CgJouzEnOueRLwP631RQCtdXIN12hp5lyzBuqVf+wLnKvB+ixOa70eSL/KITcDn2nDVsBPKdWwOue0VVA3BhIu+zyx/GsVHqO1LgEygcAaqc7yzLney03E+B/ZkVV6zUqpzkCo1vqbmizMisz5PocD4UqpTUqprUqpoTVWnXWYc80vA/copRKB74BHaqY0m6nqv/dK2Wpz24p6xn+eJ2jOMY7C7GtRSt0DRAMDrFqR9V31mpVSJuBdYHxNFVQDzPk+O2MMf8Rg/Na0QSkVqbXOsHJt1mLONd8FzNVav62U6gXMK7/mMuuXZxMWzy5b9agTgdDLPm/CX38dunSMUsoZ41emq/26Yc/MuV6UUoOAF4ARWuvCGqrNWiq7Zh8gElirlIrHGMtb4eBvKJr7c/211rpYa30KOIIR3I7KnGueCHwBoLXeArhjLF5UW5n1770qbBXUO4DWSqnmSilXjDcLV/zpmBXAuPKPRwJrdPlIvQOq9HrLhwFmYoS0o49bQiXXrLXO1FrX11qHaa3DMMblR2it42xTrkWY83P9FcYbxyil6mMMhZys0Soty5xrPgPEAiil2mEEdUqNVlmzVgD3ls/+6Alkaq3PV6tFG75zOgw4ivGO8QvlX3sV4x8rGN/MJcBxYDvQwtbv9lr5elcBScDu8scKW9ds7Wv+07FrcfBZH2Z+nxXwDnAQ2AeMtnXNNXDN7YFNGDNCdgNDbF1zNa/3c+A8UIzRe54IPAg8eNn3+H/lfx/7LPFzLbeQCyGEnZM7E4UQws5JUAshhJ2ToBZCCDsnQS2EEHZOgloIIeycBLUQQtg5CWohhLBz/x9iEaoXVqkDNAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 3\n",
    "import sympy as sym\n",
    "x = sym.Symbol('x')\n",
    "psi = [sym.sin(sym.pi*(i+1)*x) for i in range(N+1)]\n",
    "f = 10*(x-1)**2 - 1\n",
    "Omega = [0, 1]\n",
    "u, c = least_squares(f, psi, Omega)\n",
    "comparison_plot(f, u, Omega)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Figure](#fem:approx:global:fig:parabola:sine1) (left) shows the oscillatory approximation\n",
    "of $\\sum_{j=0}^Nc_j\\sin ((j+1)\\pi x)$ when $N=3$.\n",
    "Changing $N$ to 11 improves the approximation considerably, see\n",
    "[Figure](#fem:approx:global:fig:parabola:sine1) (right).\n",
    "\n",
    "<!-- dom:FIGURE: [fig/parabola_ls_sines4_12.png, width=800]  Best approximation of a parabola by a sum of 3 (left) and 11 (right) sine functions.  <div id=\"fem:approx:global:fig:parabola:sine1\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:fig:parabola:sine1\"></div>\n",
    "\n",
    "<p>Best approximation of a parabola by a sum of 3 (left) and 11 (right) sine functions.</p>\n",
    "<img src=\"fig/parabola_ls_sines4_12.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "There is an error $f(0)-u(0)=9$ at $x=0$ in [Figure](#fem:approx:global:fig:parabola:sine1) regardless of how large $N$ is, because all ${\\psi}_i(0)=0$ and hence\n",
    "$u(0)=0$. We may help the approximation to be correct at $x=0$ by\n",
    "seeking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto21\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "u(x) = f(0) + \\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x)\n",
    "{\\thinspace .}\n",
    "\\label{_auto21} \\tag{49}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, this adjustment introduces a new problem at $x=1$ since\n",
    "we now get an error $f(1)-u(1)=f(1)-0=-1$ at this point. A more\n",
    "clever adjustment is to replace the $f(0)$ term by a term that\n",
    "is $f(0)$ at $x=0$ and $f(1)$ at $x=1$. A simple linear combination\n",
    "$f(0)(1-x) + xf(1)$ does the job:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto22\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "u(x) = f(0)(1-x) + xf(1) + \\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x)\n",
    "{\\thinspace .}\n",
    "\\label{_auto22} \\tag{50}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adjustment of $u$ alters the linear system slightly. In the general\n",
    "case, we set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "u(x) = B(x) +  \\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the linear system becomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sum_{j\\in{\\mathcal{I}_s}}({\\psi}_i,{\\psi}_j)c_j = (f-B,{\\psi}_i),\\quad i\\in{\\mathcal{I}_s}{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The calculations can still utilize the `least_squares` or\n",
    "`least_squares_orth` functions, but solve for $u-b$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...evaluating matrix... (0,0)\n",
      "(1,1)\n",
      "(2,2)\n",
      "(3,3)\n",
      "A:\n",
      " [1/2, 1/2, 1/2, 1/2] \n",
      "b:\n",
      " [-40/pi**3 + 9/pi, 9/(2*pi), -40/(27*pi**3) + 3/pi, 9/(4*pi)]\n",
      "coeff: [-80/pi**3 + 18/pi, 9/pi, -80/(27*pi**3) + 6/pi, 9/(2*pi)]\n",
      "approximation: (-80/pi**3 + 18/pi)*sin(pi*x) + 9*sin(2*pi*x)/pi + (-80/(27*pi**3) + 6/pi)*sin(3*pi*x) + 9*sin(4*pi*x)/(2*pi)\n"
     ]
    }
   ],
   "source": [
    "from src.approx1D import least_squares_orth\n",
    "\n",
    "f0 = 0;  f1 = -1\n",
    "b = f0*(1-x) + x*f1\n",
    "u_sum, c = least_squares_orth(f-b, psi, Omega)\n",
    "u = B + u_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Figure](#fem:approx:global:fig:parabola:sine2) shows the result\n",
    "of the technique for\n",
    "ensuring the right boundary values. Even 3 sines can now adjust the\n",
    "$f(0)(1-x) + xf(1)$ term such that $u$ approximates the parabola really\n",
    "well, at least visually.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/parabola_ls_sines4_12_wfterm.png, width=800]  Best approximation of a parabola by a sum of 3 (left) and 11 (right) sine functions with a boundary term.  <div id=\"fem:approx:global:fig:parabola:sine2\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:fig:parabola:sine2\"></div>\n",
    "\n",
    "<p>Best approximation of a parabola by a sum of 3 (left) and 11 (right) sine functions with a boundary term.</p>\n",
    "<img src=\"fig/parabola_ls_sines4_12_wfterm.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "## Orthogonal basis functions\n",
    "<div id=\"fem:approx:global:orth\"></div>\n",
    "\n",
    "The choice of sine functions ${\\psi}_i(x)=\\sin ((i+1)\\pi x)$ has a great\n",
    "computational advantage: on $\\Omega=[0,1]$ these basis functions are\n",
    "*orthogonal*, implying that $A_{i,j}=0$ if $i\\neq j$. In fact, when\n",
    "$V$ contains the basic functions used in a Fourier series expansion,\n",
    "the approximation method derived in the section [Approximation principles](#fem:approx:global) results in the classical Fourier series for $f(x)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With orthogonal basis functions we can make the\n",
    "`least_squares` function (much) more efficient since we know that\n",
    "the matrix is diagonal and only the diagonal elements need to be computed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares_orth(f, psi, Omega):\n",
    "    N = len(psi) - 1\n",
    "    A = [0]*(N+1)\n",
    "    b = [0]*(N+1)\n",
    "    x = sym.Symbol('x')\n",
    "    for i in range(N+1):\n",
    "        A[i] = sym.integrate(psi[i]**2, (x, Omega[0], Omega[1]))\n",
    "        b[i] = sym.integrate(psi[i]*f,  (x, Omega[0], Omega[1]))\n",
    "    c = [b[i]/A[i] for i in range(len(b))]\n",
    "    u = 0\n",
    "    for i in range(len(psi)):\n",
    "        u += c[i]*psi[i]\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned in the section [Implementation of the least squares method](#fem:approx:global:LS:code), symbolic integration\n",
    "may fail or take a very long time. It is therefore natural to extend the\n",
    "implementation above with a version where we can choose between symbolic\n",
    "and numerical integration and fall back on the latter if the former\n",
    "fails:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares_orth(f, psi, Omega, symbolic=True):\n",
    "    N = len(psi) - 1\n",
    "    A = [0]*(N+1)       # plain list to hold symbolic expressions\n",
    "    b = [0]*(N+1)\n",
    "    x = sym.Symbol('x')\n",
    "    for i in range(N+1):\n",
    "        # Diagonal matrix term\n",
    "        A[i] = sym.integrate(psi[i]**2, (x, Omega[0], Omega[1]))\n",
    "\n",
    "        # Right-hand side term\n",
    "        integrand = psi[i]*f\n",
    "        if symbolic:\n",
    "            I = sym.integrate(integrand,  (x, Omega[0], Omega[1]))\n",
    "        if not symbolic or isinstance(I, sym.Integral):\n",
    "            print(('numerical integration of', integrand))\n",
    "            integrand = sym.lambdify([x], integrand, 'mpmath')\n",
    "            I = mpmath.quad(integrand, [Omega[0], Omega[1]])\n",
    "        b[i] = I\n",
    "    c = [b[i]/A[i] for i in range(len(b))]\n",
    "    u = 0\n",
    "    u = sum(c[i]*psi[i] for i in range(len(psi)))\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function is found in the file [`approx1D.py`](src/approx1D.py). Observe that\n",
    "we here assume that\n",
    "$\\int_\\Omega{\\varphi}_i^2{\\, \\mathrm{d}x}$ can always be symbolically computed,\n",
    "which is not an unreasonable assumption\n",
    "when the basis functions are orthogonal, but there is no guarantee,\n",
    "so an improved version of the function above would implement\n",
    "numerical integration also for the `A[i,i]` term.\n",
    "\n",
    "## Numerical computations\n",
    "\n",
    "Sometimes the basis functions ${\\psi}_i$ and/or the function $f$\n",
    "have a nature that makes symbolic integration CPU-time\n",
    "consuming or impossible.\n",
    "Even though we implemented a fallback on numerical integration\n",
    "of $\\int f{\\varphi}_i {\\, \\mathrm{d}x}$, considerable time might still be required\n",
    "by `sympy` just by *attempting* to integrate symbolically.\n",
    "Therefore, it will be handy to have a function for fast\n",
    "*numerical integration and numerical solution\n",
    "of the linear system*. Below is such a method. It requires\n",
    "Python functions `f(x)` and `psi(x,i)` for $f(x)$ and ${\\psi}_i(x)$\n",
    "as input. The output is a mesh function\n",
    "with values `u` on the mesh with points in the array `x`.\n",
    "Three numerical integration methods are offered:\n",
    "`scipy.integrate.quad` (precision set to $10^{-8}$),\n",
    "`mpmath.quad` (about machine precision), and a Trapezoidal\n",
    "rule based on the points in `x` (unknown accuracy, but\n",
    "increasing with the number of mesh points in `x`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares_numerical(f, psi, N, x,\n",
    "                            integration_method='scipy',\n",
    "                            orthogonal_basis=False):\n",
    "    import scipy.integrate\n",
    "    A = np.zeros((N+1, N+1))\n",
    "    b = np.zeros(N+1)\n",
    "    Omega = [x[0], x[-1]]\n",
    "    dx = x[1] - x[0]       # assume uniform partition\n",
    "\n",
    "    for i in range(N+1):\n",
    "        j_limit = i+1 if orthogonal_basis else N+1\n",
    "        for j in range(i, j_limit):\n",
    "            print(('(%d,%d)' % (i, j)))\n",
    "            if integration_method == 'scipy':\n",
    "                A_ij = scipy.integrate.quad(\n",
    "                    lambda x: psi(x,i)*psi(x,j),\n",
    "                    Omega[0], Omega[1], epsabs=1E-9, epsrel=1E-9)[0]\n",
    "            elif integration_method == 'sympy':\n",
    "                A_ij = mpmath.quad(\n",
    "                    lambda x: psi(x,i)*psi(x,j),\n",
    "                    [Omega[0], Omega[1]])\n",
    "            else:\n",
    "                values = psi(x,i)*psi(x,j)\n",
    "                A_ij = trapezoidal(values, dx)\n",
    "            A[i,j] = A[j,i] = A_ij\n",
    "\n",
    "        if integration_method == 'scipy':\n",
    "            b_i = scipy.integrate.quad(\n",
    "                lambda x: f(x)*psi(x,i), Omega[0], Omega[1],\n",
    "                epsabs=1E-9, epsrel=1E-9)[0]\n",
    "        elif integration_method == 'sympy':\n",
    "            b_i = mpmath.quad(\n",
    "                lambda x: f(x)*psi(x,i), [Omega[0], Omega[1]])\n",
    "        else:\n",
    "            values = f(x)*psi(x,i)\n",
    "            b_i = trapezoidal(values, dx)\n",
    "        b[i] = b_i\n",
    "\n",
    "    c = b/np.diag(A) if orthogonal_basis else np.linalg.solve(A, b)\n",
    "    u = sum(c[i]*psi(x, i) for i in range(N+1))\n",
    "    return u, c\n",
    "\n",
    "def trapezoidal(values, dx):\n",
    "    \"\"\"Integrate values by the Trapezoidal rule (mesh size dx).\"\"\"\n",
    "    return dx*(np.sum(values) - 0.5*values[0] - 0.5*values[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example on calling the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0,0)\n",
      "(1,1)\n",
      "(2,2)\n",
      "(3,3)\n",
      "(4,4)\n",
      "(5,5)\n",
      "(6,6)\n",
      "(7,7)\n",
      "(8,8)\n",
      "(9,9)\n",
      "(10,10)\n",
      "(11,11)\n",
      "(12,12)\n",
      "(13,13)\n",
      "(14,14)\n",
      "(15,15)\n",
      "(16,16)\n",
      "(17,17)\n",
      "(18,18)\n",
      "(19,19)\n",
      "(20,20)\n"
     ]
    }
   ],
   "source": [
    "def psi(x, i):\n",
    "    return np.sin((i+1)*x)\n",
    "\n",
    "x = np.linspace(0, 2*pi, 501)\n",
    "N = 20\n",
    "u, c = least_squares_numerical(lambda x: np.tanh(x-np.pi), psi, N, x,\n",
    "                               orthogonal_basis=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark.**\n",
    "The `scipy.integrate.quad` integrator is usually much faster than\n",
    "`mpmath.quad`.\n",
    "\n",
    "# Interpolation\n",
    "\n",
    "## The interpolation (or collocation) principle\n",
    "<div id=\"fem:approx:global:interp\"></div>\n",
    "\n",
    "\n",
    "The principle of minimizing the distance between $u$ and $f$ is\n",
    "an intuitive way of computing a best approximation $u\\in V$ to $f$.\n",
    "However, there are other approaches as well.\n",
    "One is to demand that $u(x_{i}) = f(x_{i})$ at some selected points\n",
    "$x_{i}$, $i\\in{\\mathcal{I}_s}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto24\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "u(x_{i}) = \\sum_{j\\in{\\mathcal{I}_s}} c_j {\\psi}_j(x_{i}) = f(x_{i}),\n",
    "\\quad i\\in{\\mathcal{I}_s}{\\thinspace .}  \\label{_auto24} \\tag{52}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recognize that the equation $\\sum_j c_j {\\psi}_j(x_{i}) = f(x_{i})$\n",
    "is actually a linear system with $N+1$ unknown coefficients $\\left\\{ {c}_j \\right\\}_{j\\in{\\mathcal{I}_s}}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto25\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\sum_{j\\in{\\mathcal{I}_s}} A_{i,j}c_j = b_i,\\quad i\\in{\\mathcal{I}_s},\n",
    "\\label{_auto25} \\tag{53}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with coefficient matrix and right-hand side vector given by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto26\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "A_{i,j} = {\\psi}_j(x_{i}),\n",
    "\\label{_auto26} \\tag{54}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto27\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "b_i = f(x_{i}){\\thinspace .}   \\label{_auto27} \\tag{55}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the coefficient matrix is not symmetric because\n",
    "${\\psi}_j(x_{i})\\neq {\\psi}_i(x_{j})$ in general.\n",
    "The method is often referred to as an *interpolation method*\n",
    "since some point values of $f$ are given ($f(x_{i})$) and we\n",
    "fit a continuous function $u$ that goes through the $f(x_{i})$ points.\n",
    "In this case the $x_{i}$ points are called *interpolation points*.\n",
    "When the same approach is used to approximate differential equations,\n",
    "one usually applies the name *collocation method* and\n",
    "$x_{i}$ are known as *collocation points*.\n",
    "\n",
    "Given $f$  as a `sympy` symbolic expression `f`, $\\left\\{ {{\\psi}}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$\n",
    "as a list `psi`, and a set of points $\\left\\{ {x}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$  as a list or array\n",
    "`points`, the following Python function sets up and solves the matrix system\n",
    "for the coefficients $\\left\\{ {c}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def interpolation(f, psi, points):\n",
    "    N = len(psi) - 1\n",
    "    A = sym.zeros(N+1, N+1)\n",
    "    b = sym.zeros(N+1, 1)\n",
    "    psi_sym = psi  # save symbolic expression\n",
    "    x = sym.Symbol('x')\n",
    "    psi = [sym.lambdify([x], psi[i], 'mpmath') for i in range(N+1)]\n",
    "    f = sym.lambdify([x], f, 'mpmath')\n",
    "    for i in range(N+1):\n",
    "        for j in range(N+1):\n",
    "            A[i,j] = psi[j](points[i])\n",
    "        b[i,0] = f(points[i])\n",
    "    c = A.LUsolve(b)\n",
    "    # c is a sympy Matrix object, turn to list\n",
    "    c = [sym.simplify(c[i,0]) for i in range(c.shape[0])]\n",
    "    u = sym.simplify(sum(c[i]*psi_sym[i] for i in range(N+1)))\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `interpolation` function is a part of the [`approx1D`](src/approx1D.py)\n",
    "module.\n",
    "\n",
    "\n",
    "We found it convenient in the above function to turn the expressions `f` and\n",
    "`psi` into ordinary Python functions of `x`, which can be called with\n",
    "`float` values in the list `points` when building the matrix and\n",
    "the right-hand side. The alternative is to use the `subs` method\n",
    "to substitute the `x` variable in an expression by an element from\n",
    "the `points` list. The following session illustrates both approaches\n",
    "in a simple setting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle 0.25$"
      ],
      "text/plain": [
       "0.250000000000000"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = sym.Symbol('x')\n",
    "e = x**2              # symbolic expression involving x\n",
    "p = 0.5               # a value of x\n",
    "v = e.subs(x, p)      # evaluate e for x=p\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sympy.core.numbers.Float"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = lambdify([x], e)  # make Python function of e\n",
    "type(e)\n",
    "function\n",
    "v = e(p)              # evaluate e(x) for x=p\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "float"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice feature of the interpolation or collocation method is that it\n",
    "avoids computing integrals. However, one has to decide on the location\n",
    "of the $x_{i}$ points.  A simple, yet common choice, is to\n",
    "distribute them uniformly throughout the unit interval.\n",
    "\n",
    "### Example\n",
    "\n",
    "Let us illustrate the interpolation method by approximating\n",
    "our parabola $f(x)=10(x-1)^2-1$ by a linear function on $\\Omega=[1,2]$,\n",
    "using two collocation points $x_0=1+1/3$ and $x_1=1+2/3$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deVxVdf7H8deXTREUUdwRxQ0VUEHc99wrLW1zT510ypxsyqaa3zQt02JumVmZ5TLtlmmZGbjv+1ZubCoiLiigIDuX+/39ccgcc0vu5dwLn+fjwUO499zv+RzBt4dzv+f7UVprhBBCOC8XswsQQghRPBLkQgjh5CTIhRDCyUmQCyGEk5MgF0IIJ+dmxk79/Px0/fr1zdi1EEI4rb1796Zoratd+7gpQV6/fn327Nljxq6FEMJpKaVOXu9xubQihBBOToJcCCGcnAS5EEI4OVOukV9PQUEBSUlJ5Obmml2KuIXy5cvj7++Pu7u72aUIIXCgIE9KSqJixYrUr18fpZTZ5Ygb0FqTmppKUlISgYGBZpcjhMCBLq3k5uZStWpVCXEHp5SiatWq8puTEA7EYYIckBB3EvJ9EsKxOFSQCyFEqZWfDT8/DzkXbT60BLkDWr58OVOmTLHJWLNmzSI7O/vK13fffTeXLl2yydhCiNtkyYPFw2HXPDi12+bDS5DbmcVi+dOvGThwIC+88IJN9n9tkK9cuZLKlSvbZGwhxG0otMCSsXBsHQyYDU362HwXEuTXuP/++2ndujXBwcHMmzcPAG9vb5599lnCw8Pp2bMnFy5cAKB79+48/fTTdOzYkZCQEHbt2gXAK6+8wvjx4+nTpw+jRo0iNzeXMWPGEBoaSlhYGOvXrwdg5syZjB07FoCDBw8SEhJCdnY2ixYtYuLEiQCMHj2aJ554gh49etCgQQM2btzI2LFjadasGaNHj75S9xNPPEFERATBwcG8/PLLAMyePZszZ87Qo0cPevToARjLI6SkpFzZf0hICCEhIcyaNQuAhIQEmjVrxrhx4wgODqZPnz7k5OTY869ciNLLaoUfnoToFdBvCoSPtMtuHGb64dVe/fEwR85k2HTM5rUr8fKA4Ftut2DBAqpUqUJOTg5t2rThgQceICsri/DwcGbMmMFrr73Gq6++ypw5cwDIyspi27ZtbNq0ibFjx3Lo0CEA9u7dy5YtW/D09GTGjBmAEdbR0dH06dOH2NhYnn76abp3786yZct44403+Oijj6hQocIfarp48SLr1q1j+fLlDBgwgK1bt/LJJ5/Qpk0bDhw4QKtWrXjjjTeoUqUKhYWF9OzZk19//ZWnnnqKmTNnsn79evz8/P5nzL1797Jw4UJ27tyJ1pp27drRrVs3fH19iYuL46uvvuLjjz/m4Ycf5rvvvmPEiBHF/RYIUbZoDSufhV+/hh7/B+2fsNuu5Iz8GrNnz6Zly5a0b9+eU6dOERcXh4uLC4888ggAI0aMYMuWLVe2Hzp0KABdu3YlIyPjyvXngQMH4unpCcCWLVsYOdL4n7hp06bUq1eP2NhYXFxcWLRoESNHjqRbt2506tTpujUNGDAApRShoaHUqFGD0NBQXFxcCA4OJiEhAYBvvvmG8PBwwsLCOHz4MEeOHLnpcW7ZsoVBgwbh5eWFt7c3gwcPZvPmzQAEBgbSqlUrAFq3bn1lH0KI26Q1RL4IexZA579D1+fsujuHPCO/nTNne9iwYQNr1qxh+/btVKhQge7du193vvTV0++unYr329deXl5XHrtZg+u4uDi8vb05c+bMDbcpV64cAC4uLlc+/+1ri8XCiRMnmD59Ort378bX15fRo0ffcp73zWq6eh+urq5yaUWIP0NrWPsq7PwQ2j0BPV8GO0/ZlTPyq6Snp+Pr60uFChWIjo5mx44dAFitVpYsWQLAl19+SefOna+8ZvHixYBxhuvj44OPj88fxu3atStffPEFALGxsSQmJhIUFER6ejqTJk1i06ZNpKamXtnHn5WRkYGXlxc+Pj4kJyfz888/X3muYsWKXL58+bo1ff/992RnZ5OVlcWyZcvo0qXLHe1fCHGVjVNhyzsQMRb6vWX3EAcHPSM3S79+/Zg7dy4tWrQgKCiI9u3bA8bZ9eHDh2ndujU+Pj5XwhvA19eXjh07kpGRwYIFC6477oQJE3j88ccJDQ3Fzc2NRYsWUa5cOZ544gkmTJhAkyZNmD9/Pj169KBr165/uu6WLVsSFhZGcHAwDRo0+J9LNOPHj6d///7UqlXrypusAOHh4YwePZq2bdsC8NhjjxEWFiaXUYQoji2zYMOb0Go43D2jREIcQN3sV2x7iYiI0Nc2ljh69CjNmjUr8Vpuh7e3N5mZmX94vHv37kyfPp2IiAgTqjKXI3+/hDDFjrkQ+TyEPACDPwYXV5vvQim1V2v9h8CRSytCCFFcexYaId70Xhj0kV1C/Gbk0sptuN7ZOBhvjgohyrgDX8GKv0PjPvDgQnAt+eWd5YxcCCHu1KHv4IcJ0KAbPPwZuHmYUoYEuRBC3ImjK+C7cVC3PQz5EtzLm1aKTYJcKfV3pdRhpdQhpdRXSinzjkgIIewtJhK+HQ21w2D4N+DhdcuX2FOxg1wpVQd4CojQWocArsCQ4o4rhBAOKSYSvhkJNUNgxHdQrqLZFdns0oob4KmUcgMqADe+TbEM2rBhA9u2bTO7DCFEcf0W4jWCYeQy8HSMlUSLHeRa69PAdCAROAuka61XXbudUmq8UmqPUmrPb6sHlhUS5EKUArFRRohXb14U4r5mV3SFLS6t+AL3AYFAbcBLKfWHpfK01vO01hFa64hq1aoVd7d28fnnn9O2bVtatWrFX//6V06ePEnjxo1JSUnBarXSpUsXVq0y/o+63nK3AJGRkYSHh9OyZUt69uxJQkICc+fO5Z133qFVq1ZXFqYSQjiR2ChYPMII8VHfO1SIg23mkfcCTmitLwAopZYCHYHP73jEn1+AcwdtUNpVaoZC/xt33Tl69CiLFy9m69atuLu7M2HCBDZu3Mjzzz/P448/Trt27WjevDl9+hiLwl9vuVur1cq4cePYtGkTgYGBpKWlUaVKFR5//HG8vb2ZPHmybY9JCGF/Dh7iYJsgTwTaK6UqADlAT2DPzV/ieNauXcvevXtp06YNADk5OVSvXp1XXnmFb7/9lrlz53LgwIEr28+ePZtly5YBXFnu9sKFC3Tt2pXAwEAAqlSpUvIHIoSwndhVRSHezGFDHGwQ5FrrnUqpJcA+wALsB+bd/FW3cJMzZ3vRWvPoo4/y1ltv/c/j2dnZJCUlAcYdnhUrVrzhcrdaa+kwL0RpEbvK6LNZvRmM+sFhQxxsNGtFa/2y1rqp1jpEaz1Sa51ni3FLUs+ePVmyZAnnz58HIC0tjZMnT/L8888zfPhwXnvtNcaNGwfceLnbDh06sHHjRk6cOHFlDLjxUrJCCAflRCEOcmfnFc2bN+f111+nT58+tGjRgt69e5OQkMDu3buvhLmHhwcLFy6kX79+WCwWWrRowUsvvXRludtq1aoxb948Bg8eTMuWLa90FRowYADLli2TNzuFcAaxUb+H+EjHvZxyNVnGVtwR+X6JUunIcqPj/W/zxCs41vtcsoytEELczMElRbfdtzIupzhYiN+MBLkQQhz4EpaOg7rtHOqOzdvlUEFuxmUe8efJ90mUKnsWwvcTILArjFjiEGun/FkOE+Tly5cnNTVVQsLBaa1JTU2lfHlZ4FKUAjs/ghVPQ+PeMHSx6asY3imH6RDk7+9PUlISZW0dFmdUvnx5/P39zS5DiOLZOhtWv2S0Z3twAbiVM7uiO+YwQe7u7n7ljkghhLCrTdNg3esQPMholGxCezZbcpggF0IIu9Ma1r9hBHmLIXDf++Dq/DHo/EcghBC3Q2tY9S/YPgfCR8G9s0q82729SJALIUo/ayH8OAn2fwZt/wr9poCLw8z1KDYJciFE6WbJN+aIH/keuv4DevwTStnidhLkQojSKz/b6OoTvwb6vA4d/2Z2RXZRen63EEKIq+Wmw+eDIX4tDJhteognpGQx+dtfSMvKt/nYckYuhCh9Mi8YIX7+qDFHPGSwaaUkZ+Qye20ci3efwt3VhXta1KJHUHWb7kOCXAhRuqSfhk/vg/QkGPqVcdemGWVkFzB30zEWbj2BpVAztG0Af7urEdUr2f6uaAlyIUTpkXoMPr0fci/ByKVQr2OJl5CTX8iibQl8uCGejFwL97WqzTO9m1Cvqv1u/5cgF0KUDucOwWeDQBfCoz8ay9GWoIJCK9/sOcW7a+I4fzmPHkHVmNw3iODaPnbftwS5EML5JWyBr4YZi16NWgHVgkps11ar5qeDZ5mxKoaE1Gxa1/PlvaFhtGtQtcRqkCAXQji3oz/Ckr+Abz0YsRQq1y2R3Wqt2RSXwtTIaA6fySCoRkU+GRVBz2bVS7wJuwS5EMJ57VkAPz0LdVrDsG9KrKvPvsSLTI2MZsfxNPx9PZn5cEvua1UHVxdzbjSSIBdCOB+tYeNU2PAmNO4DDy0qkbXE45IvMy0qhlVHkqnq5cErA5oztF0A5dzMXbNFglwI4VyshbDyOdgzH1oOg4Gz7b4M7elLOcxaHct3+5Ko4OHGM72bMLZzIN7lHCNCHaMKIYS4HQW5xropR5dDp0nQ61W7rpuSmpnHBxuO8dn2kwCM7RTIhB6NqOLlYbd93gmbBLlSqjLwCRACaGCs1nq7LcYWQgjAuOX+6+GQsBn6vgkdnrTbrjLzLMzffIKPNx8nO9/Cg639mdSrCXUqe9ptn8VhqzPyd4FIrfWDSikPoIKNxhVCCLicDJ8/ABeOGh19Wjxsl93kWQr5cmcic9bFk5qVT7/gmkzu24RG1R27IXOxg1wpVQnoCowG0FrnA7ZfFUYIUTZdiIEvHoSsVBi2GBr1svkuCq2a7/efZubqWE5fyqFDg6r8o18QYQG+Nt+XPdjijLwBcAFYqJRqCewFJmmts67eSCk1HhgPEBAQYIPdCiFKvYSt8PVQcC0Ho1dAnXCbDq+1Zs3R80yLiiY2OZOQOpWY8kAonRv5lfhc8OJQWuviDaBUBLAD6KS13qmUehfI0Fq/dKPXRERE6D179hRrv0KIUu7gEvj+CfCtD8OXGDf82NDO46m8HRnNvsRLBPp58WyfJtwdUgsXk+aC3w6l1F6tdcS1j9vijDwJSNJa7yz6egnwgg3GFUKURVrD1lmw5hWo1wmGfAGetrvEcfhMOtOiYtgQc4Ealcrx1uBQHmztj7ur87ZnKHaQa63PKaVOKaWCtNYxQE/gSPFLE0KUOYUWWDkZ9i6EkAfh/g/ArZxNhk5IyWLm6liW/3IGH093XuzflEc71qe8u/M3YLbVrJW/AV8UzVg5Doyx0bhCiLIiLxOWjIG4VdD5GbjrJZs0SD6fkcvsdXF8vesUbq6KCd0b8tduDfHxtO9NRCXJJkGutT4A/OG6jRBC3JbLyfDlQ3DuINz7DkSMLfaQ6TkFfLTxGAuKGjsMaVuXp+5qbJfGDmaTOzuFEOY6Hw1fPATZqTD0a2jSt1jD5eQX8t/tCXy44RjpOQUMbGk0dqjvZ/+1WMwiQS6EMM+xdfDNaHAvD2N+gtphdzyUpdDKN3uSeHdtLMkZeXQPqsZzJdTYwWwS5EIIc+z+BFb+A6o1hWFfQ+U7u7/EatX8fOgcM1bFcDwli/CAyrw7JIz2JdjYwWwS5EKIklVogah/wq6PoHFfeHA+lPvzt8Brrdkcl8LUqGgOnc6gSQ1vPh4VQS8TGjuYTYJcCFFyctNhyViIXwMdJkLv18Dlz0//2594kamRMWw/nkqdyp7MeKgl94eZ19jBbBLkQoiScTEBvhwCqXEw4F1oPfpPDxF/3mjsEHXYaOzw8oDmDHOAxg5mkyAXQthf4g74ehhYLUZfzQbd/tTLr23s8PdeTfhLF8dp7GA2+VsQQtjXL4th+UTwqWv01fRrdNsvTcvK54P18Xy64yRoGNMpkAndG1LV2zZ3e5YWEuRCCPuwWmH967B5BtTvAg9/etvNkbPyLMzfcoJ5m4zGDg+E+zOpV2P8faXVwfVIkAshbC83A5aOh9ifIXwU3D0D3G7dHi3PUshXOxN5r6ixQ9/gGkzuE0TjGo7d2MFsEuRCCNtKPQZfDYXUeOg/DdqOu2VfzUKr5ocDRmOHpIs5tG9QhY/7NSXcSRo7mE2CXAhhO3Fr4LuxoFxh1PcQ2PWmm2utWXv0PNOiYohJvkxw7Uq8OSiULo2dq7GD2STIhRDFpzVsm22sIV69OQz58paNIHadSOPtyGj2nrxI/aoVeG9oGPeEOnZjB0clQS6EKJ78bFj+Nzi0BJrfb6wh7nHjBaqOnMlgWlQ062MuUL1iOd4YFMLDEXWdurGD2STIhRB37tIpY374uYPG+uFdnr3h9fCTqUZjhx8OnKFSeTde6N+URzvUx9OjbN/MYwsS5EKIO3NyGyweCZY8Y/nZoH7X3ez85VzeWxvPV7sSf2/s0LUhPhVKT2MHs0mQCyH+HK1h50ew6v+MxshDvoJqTf6wWXpOAfM2HWPBlgTyC60MaVOXp3o2pkYpbOxgNglyIcTty8uEHycZ18Ob9IdBc8Gz8v9skltQyH+3JfBBUWOHAUWNHQJLcWMHs0mQCyFuT0o8LB4BKTHG9fDOz/xPT01LoZVv9ybx7po4zmXk0q2J0dghpE7pb+xgNglyIcStHVkO308w7s4csRQa9rjylNZGY4fpUUZjh7CAyrzzSCs6NCw7jR3MJkEuhLixQgusfdWYI1473FgvpXLdK09viUvh7choDp5Op3F1b+aNbE3v5jXkZp4SJkEuhLi+zPNGE4iEzUZX+35TwM1YdfDAqUtMjYxm2zGjscP0h1oyqAw3djCbBLkQ4o8Sd8K3j0LORbh/LrQaChiNHaZHxRJ5+BxVvDz4973NGd5eGjuYzWZBrpRyBfYAp7XW99pqXCFECdIatr8Pa14GH394bA3UDOXMpRxmrYllyd4kPN1debpXYx7r0kAaOzgIW34XJgFHgUo2HFMIUVKy04w3NGN/hqB74P73uWj14oOfjvDf7UZjh9EdA3myhzR2cDQ2CXKllD9wD/AG8IwtxhRClKBTu+DbMZCZDP2mkNXqMRZsTWDepl1k5VsYHO7P09LYwWHZ6ox8FvAP4IarvyulxgPjAQICAmy0WyFEsVitsP09WPsaVKpDwehIvkzy473pG0jJzKdP8xpM7htEE2ns4NCKHeRKqXuB81rrvUqp7jfaTms9D5gHEBERoYu7XyFEMWWlwvePQ9wqdLOB/BT4IlO+OkfSxWTaBVZh3ihp7OAsbHFG3gkYqJS6GygPVFJKfa61HmGDsYUQ9nByO3z3F3TWBWLD/82kYxFE7z9BcO1KvDEolK7S2MGpFDvItdYvAi8CFJ2RT5YQF8JBWa2wdRase51cb39e8pnBt9uqUr+qVRo7ODGZOyREWZFx1riUcnwDOyt05y8XRuBZ0ZfX72/MI22ksYMzs2mQa603ABtsOaYQwgaOrqDwh4lY8nJ4peAxfsruzYR+jRjTMVAaO5QCckYuRGmWn0XOiufx/PUzjupAnrO+RPcundgsjR1KFQlyIUqpzIS95H09Bt+cRD6yDiCp1TMs6t1cGjuUQhLkQpQyufkF7P/mDSLiZ5OpKzGn7nQG3D9UGjuUYhLkQpQSlkIrK7buo9b6Z+igf2GPZye8HvyApxrWN7s0YWcS5EI4ud8aO+z4aRFP58zBSxVwrN3rRPSbeMOO9qJ0kSAXwoltjU9hzso9PHThPV5z3UK6bzAewxfSsFqQ2aWJEiRBLoQT+uXUJaZFxaCOr+Ndj3n4uaVj7fIPfLr9A1xlNkpZI0EuhBOJP5/JjFUxbDyUwCuei3nYIwpr1ca4DF4CdVqbXZ4wiQS5EE7gbHoOs1bH8e3eU3Ryj2Nb5Y/xyT0NHSbicte/wN3T7BKFiSTIhXBgF7Py+XDjMRZtS8BD5/NZQBQdk79Ela8LQ1ZA/c5mlygcgAS5EA4oK8/Cwq0n+GjjcTLzLTzVNIuJ6dNxT46B8Eeh7xtQTtYIFwYJciEcSL7Fyte7E5m9Np6UzDz6N/Xldd8VVD0wF7yqwbBvoUkfs8sUDkaCXAgHYLVqlv9yhhmrYziVlkPbwCp83kfTdOdTkBAHrUZA39fBUxo9iD+SIBfCRFpr1secZ2pkDNHnLtO8ViU+GxFI58QPUCs/Bp+6MGIpNOppdqnCgUmQC2GS3QlpTI2MZnfCRQKqVODdIa0Y4BWDy08D4dIpaDseev4bynmbXapwcBLkQpSwo2czmB4Vw9ro81SrWI7/3B/CIyGV8Fj7L9j/OVRtBGN+hnodzC5VOAkJciFKyKm0bGaujuX7A6fxLufGc32DGNOxHhXiV8DcFyDrAnT+O3R7XuaFiz9FglwIO7twOY856+L4clciLkoxvmsDnujWkMp5Z+DboRC/Gmq1hGFfQ+0ws8sVTkiCXAg7ycgt4ONNx5m/5QR5FiuPtKnLU3c1pqa3K2x/HzZMARdX6DcF2owDV/nnKO6M/OQIYWO5BYV8tv0k72+I51J2Afe0qMWzvZvQoJo3nNoFXzwN5w9D03uh/1TwqWN2ycLJSZALYSOWQivf7Uti1po4zqbn0qWxH//o25RQfx/IuQg/Pg17F0IlfxjyFTS92+ySRSkhQS5EMWmtiTx0jmmrYjh+IYuWdSsz46GWdGzkB1rDL1/DqpcgOwU6TITuL8qUQmFTEuRCFMO2+BTejozml6R0GlX3Zu6I1vQNroFSCs7+Ciufg1M7jCVmRywx3tQUwsaKHeRKqbrAp0BNwArM01q/W9xxhXBkvyYZjR02x6VQ26c8Ux9sweCwOri5ukB2Gqx/A/YsAM8qMHAOtBoOLi5mly1KKVuckVuAZ7XW+5RSFYG9SqnVWusjNhhbCIdy7EImM1fF8tPBs/hWcOdf9zRjRPt6lHd3BWsh7F0Ea16F3EvGTJQeL8r6KMLuih3kWuuzwNmizy8rpY4CdQAJclFqnE3PYfbaOL7Zk0Q5Nxee6tmYcV0CqVi+qK1a0h5YORnO7Id6nYzZKDVDzC1alBk2vUaulKoPhAE7r/PceGA8QEBAgC13K4TdXMrO58MNRmMHq9aMbF+PiXc1ws+7nLFBxllY9x848AVUrAUPzIeQB6R7vShRNgtypZQ38B3wtNY649rntdbzgHkAERER2lb7FcIesvMtLNyawNyNx8jMszAorA5/79WEulUqGBvkZ8P2ObBlFlgLoNMk6PqcNHsQprBJkCul3DFC/Aut9VJbjCmEGfItVhbvTuTdosYOvZrV4Lm+QQTVLApoqxUOfgtrX4WM09BsIPR+Fao0MLdwUabZYtaKAuYDR7XWM4tfkhAlz2rV/PjrGWasiiUxLZu29avw0chwWter8vtGJ7dD1D/hzD5jTZQHPoF6Hc0rWogitjgj7wSMBA4qpQ4UPfZPrfVKG4wthF1prdkQc4GpUTEcPZtBs1qVWDimDd2bVDPmggOknYA1L8ORH6BibRj0EYQ+LNMJhcOwxayVLYC8syOczp6ENKZGxrArIe33xg4tauPiUvTjnJUCm6bDnvng4gbd/wkdJ4KHl7mFC3ENubNTlDnR54zGDmuOnsfPuxz/uS+YR9oE4OFWdIadl2msTrjtPSjIgrARxm31lWqbW7gQNyBBLsqMU2nZvLM6lmUHTuPtUdTYoVN9KngU/TOw5Bs39GyaajR5aDYA7vo3VGtiat1C3IoEuSj1LlzO4/318Xyx86TR2KFLAx7v1hBfLw9jA6sVDi815oNfTIB6nY3VCeu2MbVuIW6XBLkotS4XNXb4pKixw8MR/jzVszG1fIraqGkNcauNAD/3K9QIheFLoFEvuaFHOBUJclHq5BYU8vmOk7y/Pp6L2QXcE1qLZ/o0oWG1oqVjtYb4tbDhLTi9ByrXg8GfGHdkykwU4YQkyEWpYSm0snTfaWatieVMUWOH5/oG0cK/srGB1nB8Pax/C5J2gU8ADJgNrYaBq7u5xQtRDBLkwulprYk6fI5pUTEcu5BFS38fpj3Ukk6N/H7f6MQmWP8mJG43OvTc+w60GgFuHuYVLoSNSJALp7btWApvR8bwy6lLNKzmxdwR4fQNrmnczKM1JGwxmhyf3GIsanX3dAgfBW7lzC5dCJuRIBdO6WBSOlOjotkcl0Itn/JMfaAFg8OLGjtoDbFRsHkGnNoJ3jWg39vQejS4lze7dCFsToJcOJXjFzKZsTqWn349S+XrNXY49B1sfgeSD4JPXeMMPGwEuHuaXboQdiNBLpzCufRc3l0bxzd7ThmNHe5qxGNdG1CpvLtxI8++L2HLO5B2DKo2hvs/hNCH5E1MUSZIkAuHdik7nw83HmPR1t8bOzzZoxHVKpYzbqXfMR+2zTaWlK3ZAh7+FJreCy6uZpcuRImRIBcO6Q+NHVrV4e+9ixo7ZJyBNfOM5sa56RDQ0ZhG2Kin3MgjyiQJcuFQCgqtfL37FLPXxnHhch69mlVnct8gmtasBGd/haXvw6EloK3GmXeHiRDQzuyyhTCVBLlwCL81dpi5OpaTqdm0qe/Lh8PDiQioDPGrIWqOMRfc3QvaPAbtHocqgWaXLYRDkCAXptJasyH2AlMjjcYOTWtWZOHoNnQP9ET9+g2smAspsUZDh16vQutHwdPX7LKFcCgS5MI0e0+m8XZkDLtOXNXYoXYmLnumw7KvIC/DeANz0DwIHiR3YQpxAxLkosTFnLvMtKgY1hxNNho7DAhiSOWjuO95Ar7fCC7uRnC3HQf+beQNTCFuQYJclJhTadm8syaWZfuNxg4vdfdjZLmNeOycBBlJUKkO3PUvCH8UvKubXa4QTkOCXNhdSmYec9YZjR1cleat0GQGsxaPXVFgLYAG3aH/FGjSH1zlR1KIP0v+1Qi7uZxbwMebTzB/83GqFJzjw7r76ZGzGtfY01ChKrT7q3H2La3UhCgWCXJhc781dpi37iht8nawpPJWmmbvQ50HGt4F/d6EoLvlzUshbESCXNiM0dghidWrVtA5Zx1rPXZS0SMD3OtC9xeMBg6VA8wuU4hSR4JcFJvWmi27dnF83UK65tfjKQUAAA5SSURBVKzjYZdkCsuVw7Xp3cbKgw16yNonQtiRTYJcKdUPeBdwBT7RWk+xxbjCwWWlcnzjp+Tv+5oulmg6oUir0Q7d4SVcmw+E8j5mVyhEmVDsIFdKuQLvA72BJGC3Umq51vpIcccWDijnIkSv5PK+JXie2kgDColX9TjQ9BlC+o7Fz7eu2RUKUebY4oy8LRCvtT4OoJT6GrgPkCAvLbLTIPonOPI9+vgGlNVCuvbjO3UPFdsO557evY3GDkIIU9giyOsAp676Ogn4w3J0SqnxwHiAgAB5w8vhZaVC9Ao48r2xWJXVQppHLb4t6M8a2tO+cy/GdWtoNHYQQpjKFkF+vfun9R8e0HoeMA8gIiLiD88Lk2kNKXEQ+zPERMKpHaCtFFYOZFeNYUxLasbBrHoMa1uP9+9qRPWK0vtSCEdhiyBPAq6+MOoPnLHBuMLeCgsgcQfERkLMSkg7bjxeM5SCjs+wLDeM1/e6cTm5kPtb1WFWryYEVK1gbs1CiD+wRZDvBhorpQKB08AQYJgNxhX2cPkcHN8AcauNdb5z08HVAwK7QvsJFDTqw+JYmL02jvOX8+jZtCqT+wbRrFYlsysXQtxAsYNca21RSk0EojCmHy7QWh8udmXCNgpy4OQ2OLYOjq2H80Xfmgp+RoedoP7QoAdWdy9WHDzLjPkxnEzNJqKeL+8PD6dN/Srm1i+EuCWbzCPXWq8EVtpiLFFM1kI4dxBObDTC++R2KMwzzroDOkCvV4zb5GuEgosLWms2xl5gauR+jhQ1dlgwOoIeQdVRsnysEE5B7ux0dpZ8OHsATm41zrwTdxgNGQCqNzfaojW8C+p1BI//vb699+RFpkZGs/NEGnWreDLrkVYMaFkbVxcJcCGciQS5s8nLhDP7jDPtk1shaTcUZBvP+QVByANQrxPU7wyVal13iNhko7HD6iPJ+Hl78Np9wQxpE4CHm0sJHogQwlYkyB2ZtRDOH4XTe+H0HkjaCxeOGh3kUVAzBMJHGWfbAR3Bu9pNh0u6mM07q+NYuj8Jbw83nu3dhLGdA/EqJz8GQjgz+RfsKKyFkHoMkg/C2V+M0D6zHwqyjOfLV4Y6raHZvVAnAuq2Bc/KtzV0amYec9bH88WORFDwWOdAnujeiCpesoysEKWBBLkZ8i5D8mHjTclzByH5ECQfAUuO8byLO9QMhbDhRmj7R0CVBn+6d2VmnoVPNh/n403HySko5KHWdZnUqzG1K3va4aCEEGaRILcXrSHrAqTEFn3EFX3EwqWTv2/n6Qs1QiBijPFnzRCo1hTcyt3xrvMshXy+I5H318eTlpVP/5CaPNsniEbVvW1wYEIIRyNBXhzWQsg4A5cSjXC+lAgXE34P7bz037d1rwBVGxln1+Ejjel/NUOhUm2bdYkvtGqW7kti1po4Tl/KoWPDqjzfrykt697eJRghhHOSIL8RSz5knYfLyZB5zrgjMvM8ZJz+PbTTk8BquepFCirWAr9G0OIh8GsCfo2hamOjQ7yLfWaFaK1ZfSSZaVExxJ3PJLSOD28/0ILOjf3ssj8hhGMpvUGuNRTmG1PzCnKMj/xMyLkEuZeMdbWv/TznonE55PI5yEm7zqAKvKtD5XrGteuQB4zWZZUDjMd8/It1SeRO7DieytuR0exPvEQDPy8+GB5O/5CacjOPEGWIcwX51tlw5AfjLFgXGpc2rIVFn1vAagVLblFwZxuP34qLuzH7o3xl40/fQAhoD941oWKNq/6sAV7VwdUx/soOnU5nWlQMG2MvULNSeaYMDuXB1v64ucpccCHKGsdIpdvlVt5oH+biCsrV+PPK527G527ljevR7p5FHxWMOxrdiz6uDm1PX+MxJzp7TUjJYsbqWH785Qw+nu788+6mjOpQXxo7CFGGOVeQtxtvfJRByRm5zF4bx+Ldp3B3dWFij0aM69oAH09p7CBEWedcQV4GpWcXMHfTMRZuPYGlUDOsXQATpbGDEOIqEuQOKie/kEXbEvhwQzyX8yzc17I2f+/dhHpVvcwuTQjhYCTIHUxBoZVv9pzi3TVGY4e7mlZncp8gmteWxg5CiOuTIHcQVqvmp4NnmbEqhoTUbFrX82XOsHDaBkpjByHEzUmQm0xrzaa4FKZGRnP4TAZBNSryyagIejaTxg5CiNsjQW6ifYlGY4cdx9Pw9/XknUdaMrBlHWnsIIT4UyTITRBX1NhhVVFjh1cHBjOkbV3KuclccCHEnydBXoKSLmYza00cS/clUUEaOwghbEQSpASkZubx/vpjfL7jJCgY2ymQCT2ksYMQwjYkyO0oM8/C/M0n+HjzcbLzLTzY2p9JvZpQRxo7CCFsSILcDvIshXy5M5E56+JJzcqnX3BNJvdtQqPqFc0uTQhRChUryJVS04ABQD5wDBijtb5ki8KcUaFVs2z/ad5ZHcvpSzl0aFCV5/s3pZU0dhBC2FFxz8hXAy9qrS1KqbeBF4Hni1+Wc9Fas+boeaZFRRObnElInUpMeSCUzo38ZC64EMLuihXkWutVV325A3iweOU4n51FjR32JV4i0M+L94cZjR1cZC64EKKE2PIa+Vhg8Y2eVEqNB8YDBAQE2HC35jh8xmjssCHmAjUqleOtosYO7tLYQQhRwm4Z5EqpNUDN6zz1f1rrH4q2+T/AAnxxo3G01vOAeQARERH6jqp1AAkpWcxcHcvyosYOL/ZvyqMdpbGDEMI8twxyrXWvmz2vlHoUuBfoqbV22oC+lfMZucxeF8fXu07h5qp4skdDxndtKI0dhBCmK+6slX4Yb25201pn26Ykx5KeU8BHG4+xoKixw9C2AfztrkZUrySNHYQQjqG418jnAOWA1UWzM3ZorR8vdlUOICe/kP9uT+DDDcdIzyngvla1eUYaOwghHFBxZ600slUhjqKg0Mq3e5J4d20syRl59AiqxuS+QQTX9jG7NCGEuC65s7OI1apZeegsM1bFciIli9b1fJk9JIx2DaqaXZoQQtxUmQ9yrTWb41KYGhXNodMZNKnhLY0dhBBOpUwH+f7Ei0yNjGH78VTqVPZkxkMtuT9MGjsIIZxLmQzy+PNGY4eow8lU9fLglQHNGdouQBo7CCGcUpkK8tOXcpi1Opbviho7PFPU2MFbGjsIIZxYmUiwtKx83l8fz2fbTwIwplMgT0pjByFEKVGqgzwrz8L8LSeYt8lo7PBAuD9P95bGDkKI0qVUBnmepZCvdibyXlFjh77BNZjcJ4jGNaSxgxCi9ClVQV5o1fxw4DQzV8eSdDGH9g2q8Em/poQF+JpdmhBC2E2pCPLfGjtMj4ohJvkyIXUq8eagULo0lsYOQojSz+mDfNeJNN6OjGbvyYsE+nkxZ1gYd4fUksYOQogyw2mD/MiZDKZFRbO+qLHDm4NCeShCGjsIIcoepwvyk6lGY4cfDpyhUnk3XujflEc71MfTQ27mEUKUTU4V5LPXxjF7bRxurooJ3Rvy164N8akgjR2EEGWbUwV53SqeDGlbl6fuaiyNHYQQoohTBfmgMH8GhfmbXYYQQjgUeWdQCCGcnAS5EEI4OQlyIYRwchLkQgjh5CTIhRDCyUmQCyGEk5MgF0IIJydBLoQQTk5prUt+p0pdAE7e4cv9gBQbluMM5JjLBjnmsqE4x1xPa13t2gdNCfLiUErt0VpHmF1HSZJjLhvkmMsGexyzXFoRQggnJ0EuhBBOzhmDfJ7ZBZhAjrlskGMuG2x+zE53jVwIIcT/csYzciGEEFeRIBdCCCfnkEGulFqglDqvlDp0g+eVUmq2UipeKfWrUiq8pGu0tds45uFFx/qrUmqbUqplSddoa7c65qu2a6OUKlRKPVhStdnL7RyzUqq7UuqAUuqwUmpjSdZnD7fxs+2jlPpRKfVL0TGPKekabUkpVVcptV4pdbToeCZdZxubZphDBjmwCOh3k+f7A42LPsYDH5ZATfa2iJsf8wmgm9a6BfAfSsebRIu4+TGjlHIF3gaiSqKgErCImxyzUqoy8AEwUGsdDDxUQnXZ0yJu/n1+EjiitW4JdAdmKKU8SqAue7EAz2qtmwHtgSeVUs2v2camGeaQQa613gSk3WST+4BPtWEHUFkpVatkqrOPWx2z1nqb1vpi0Zc7AKfveXcb32eAvwHfAeftX5H93cYxDwOWaq0Ti7Z3+uO+jWPWQEWllAK8i7a1lERt9qC1Pqu13lf0+WXgKFDnms1smmEOGeS3oQ5w6qqvk/jjX1Rp9hfgZ7OLsDelVB1gEDDX7FpKUBPAVym1QSm1Vyk1yuyCSsAcoBlwBjgITNJaW80tyTaUUvWBMGDnNU/ZNMOcqvnyVdR1HisT8yiVUj0wgryz2bWUgFnA81rrQuNkrUxwA1oDPQFPYLtSaofWOtbcsuyqL3AAuAtoCKxWSm3WWmeYW1bxKKW8MX6bfPo6x2LTDHPWIE8C6l71tT/G/+almlKqBfAJ0F9rnWp2PSUgAvi6KMT9gLuVUhat9ffmlmVXSUCK1joLyFJKbQJaAqU5yMcAU7RxU0u8UuoE0BTYZW5Zd04p5Y4R4l9orZdeZxObZpizXlpZDowqeue3PZCutT5rdlH2pJQKAJYCI0v52dkVWutArXV9rXV9YAkwoZSHOMAPQBellJtSqgLQDuMaa2mWiPEbCEqpGkAQcNzUioqh6Fr/fOCo1nrmDTazaYY55Bm5UuorjHev/ZRSScDLgDuA1nousBK4G4gHsjH+R3dqt3HM/waqAh8UnaFanH3VuNs45lLnVsestT6qlIoEfgWswCda65tOz3R0t/F9/g+wSCl1EOOSw/Naa2de2rYTMBI4qJQ6UPTYP4EAsE+GyS36Qgjh5Jz10ooQQogiEuRCCOHkJMiFEMLJSZALIYSTkyAXQggnJ0EuhBBOToJcCCGc3P8DPAovWKrBtusAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = sym.Symbol('x')\n",
    "f = 10*(x-1)**2 - 1\n",
    "psi = [1, x]\n",
    "Omega = [1, 2]\n",
    "points = [1 + sym.Rational(1,3), 1 + sym.Rational(2,3)]\n",
    "u, c = interpolation(f, psi, points)\n",
    "comparison_plot(f, u, Omega)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting linear system becomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\left(\\begin{array}{ll}\n",
    "1 & 4/3\\\\\n",
    "1 & 5/3\\\\\n",
    "\\end{array}\\right)\n",
    "\\left(\\begin{array}{l}\n",
    "c_0\\\\\n",
    "c_1\\\\\n",
    "\\end{array}\\right)\n",
    "=\n",
    "\\left(\\begin{array}{l}\n",
    "1/9\\\\\n",
    "31/9\\\\\n",
    "\\end{array}\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with solution $c_0=-119/9$ and $c_1=10$.\n",
    "[Figure](#fem:approx:global:linear:interp:fig1) (left) shows the resulting\n",
    "approximation $u=-119/9 + 10x$.\n",
    "We can easily test other interpolation points, say $x_0=1$ and $x_1=2$.\n",
    "This changes the line quite significantly, see\n",
    "[Figure](#fem:approx:global:linear:interp:fig1) (right).\n",
    "\n",
    "<!-- dom:FIGURE: [fig/parabola_inter.png, width=800]  Approximation of a parabola by linear functions computed by two interpolation points: 4/3 and 5/3 (left) versus 1 and 2 (right).  <div id=\"fem:approx:global:linear:interp:fig1\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:linear:interp:fig1\"></div>\n",
    "\n",
    "<p>Approximation of a parabola by linear functions computed by two interpolation points: 4/3 and 5/3 (left) versus 1 and 2 (right).</p>\n",
    "<img src=\"fig/parabola_inter.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lagrange polynomials\n",
    "<div id=\"fem:approx:global:Lagrange\"></div>\n",
    "\n",
    "In the section [Fourier series](#fem:approx:global:Fourier) we explained the advantage\n",
    "of having a diagonal matrix: formulas for the coefficients\n",
    "$\\left\\{ {c}_i \\right\\}_{i\\in{\\mathcal{I}_s}}$ can then be derived by hand. For an interpolation (or\n",
    "collocation) method a diagonal matrix implies that ${\\psi}_j(x_{i})\n",
    "= 0$ if $i\\neq j$. One set of basis functions ${\\psi}_i(x)$ with this\n",
    "property is the *Lagrange interpolating polynomials*, or just\n",
    "*Lagrange polynomials*. (Although the functions are named after\n",
    "Lagrange, they were first discovered by Waring in 1779, rediscovered\n",
    "by Euler in 1783, and published by Lagrange in 1795.)  Lagrange\n",
    "polynomials are key building blocks in the finite element method, so\n",
    "familiarity with these polynomials will be required anyway.\n",
    "\n",
    "A Lagrange polynomial can be written as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:global:Lagrange:poly\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "{\\psi}_i(x) =\n",
    "\\prod_{j=0,j\\neq i}^N\n",
    "\\frac{x-x_{j}}{x_{i}-x_{j}}\n",
    "= \\frac{x-x_0}{x_{i}-x_0}\\cdots\\frac{x-x_{i-1}}{x_{i}-x_{i-1}}\\frac{x-x_{i+1}}{x_{i}-x_{i+1}}\n",
    "\\cdots\\frac{x-x_N}{x_{i}-x_N},\n",
    "\\label{fem:approx:global:Lagrange:poly} \\tag{56}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for $i\\in{\\mathcal{I}_s}$.\n",
    "We see from ([56](#fem:approx:global:Lagrange:poly)) that all the ${\\psi}_i$\n",
    "functions are polynomials of degree $N$ which have the property"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:inter:prop\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "{\\psi}_i(x_s) = \\delta_{is},\\quad \\delta_{is} =\n",
    "\\left\\lbrace\\begin{array}{ll}\n",
    "1, & i=s,\\\\\n",
    "0, & i\\neq s,\n",
    "\\end{array}\\right.\n",
    "\\label{fem:inter:prop} \\tag{57}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "when $x_s$ is an interpolation (collocation) point.\n",
    "Here we have used the *Kronecker delta* symbol $\\delta_{is}$.\n",
    "This property implies that $A$ is a diagonal matrix, i.e., $A_{i,j}=0$ for $i\\neq j$ and\n",
    "$A_{i,j}=1$ when $i=j$. The solution of the linear system is\n",
    "then simply"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto28\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "c_i = f(x_{i}),\\quad i\\in{\\mathcal{I}_s},\n",
    "\\label{_auto28} \\tag{58}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto29\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "u(x) = \\sum_{j\\in{\\mathcal{I}_s}} c_i {\\psi}_i(x) = \\sum_{j\\in{\\mathcal{I}_s}} f(x_{i}){\\psi}_i(x){\\thinspace .}   \\label{_auto29} \\tag{59}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remark however that ([57](#fem:inter:prop)) does not necessarily imply that the matrix\n",
    "obtained by the least squares or projection methods is diagonal.\n",
    "\n",
    "\n",
    "The following function computes the Lagrange interpolating polynomial\n",
    "${\\psi}_i(x)$ on the unit interval (0,1), given the interpolation points $x_{0},\\ldots,x_{N}$ in\n",
    "the list or array `points`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Lagrange_polynomial(x, i, points):\n",
    "    p = 1\n",
    "    for k in range(len(points)):\n",
    "        if k != i:\n",
    "            p *= (x - points[k])/(points[i] - points[k])\n",
    "    return p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next function computes a complete basis, ${\\psi}_0,\\ldots,{\\psi}_N$, using equidistant points throughout\n",
    "$\\Omega$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Lagrange_polynomials_01(x, N):\n",
    "    if isinstance(x, sym.Symbol):\n",
    "        h = sym.Rational(1, N-1)\n",
    "    else:\n",
    "        h = 1.0/(N-1)\n",
    "    points = [i*h for i in range(N)]\n",
    "    psi = [Lagrange_polynomial(x, i, points) for i in range(N)]\n",
    "    return psi, points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When `x` is a `sym.Symbol` object, we let the spacing between the\n",
    "interpolation points, `h`, be a `sympy` rational number, so that we\n",
    "get nice end results in the formulas for ${\\psi}_i$.  The other case,\n",
    "when `x` is a plain Python `float`, signifies numerical computing, and\n",
    "then we let `h` be a floating-point number.  Observe that the\n",
    "`Lagrange_polynomial` function works equally well in the symbolic and\n",
    "numerical case - just think of `x` being a `sym.Symbol` object or a\n",
    "Python `float`.  A little interactive session illustrates the\n",
    "difference between symbolic and numerical computing of the basis\n",
    "functions and points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = sym.Symbol('x')\n",
    "psi, points = Lagrange_polynomials_01(x, N=2)\n",
    "points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1 - x, x]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "psi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 1.0]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 0.5  # numerical computing\n",
    "psi, points = Lagrange_polynomials_01(x, N=2)\n",
    "points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.5, 0.5]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "psi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, when used symbolically, the `Lagrange_polynomials_01`\n",
    "function returns the symbolic expression for the Lagrange functions\n",
    "while when `x` is a numerical value the function returns the value of\n",
    "the basis function evaluate in `x`.  In the present example only the\n",
    "second basis function should be 1 in the mid-point while the others\n",
    "are zero according to ([57](#fem:inter:prop)).\n",
    "\n",
    "\n",
    "### Approximation of a polynomial\n",
    "\n",
    "The Galerkin or least squares methods lead to an exact approximation\n",
    "if $f$ lies in the space spanned by the basis functions. It could be\n",
    "of interest to see how the interpolation method with Lagrange\n",
    "polynomials as the basis is able to approximate a polynomial, e.g., a\n",
    "parabola. Running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = sym.Symbol('x')\n",
    "for N in 2, 4, 5, 6, 8, 10, 12:\n",
    "    f = x**2\n",
    "    psi, points = Lagrange_polynomials_01(x, N)\n",
    "    u = interpolation(f, psi, points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "shows the result that up to `N=4` we achieve an exact approximation,\n",
    "and then round-off errors start to grow, such that\n",
    "`N=15` leads to a 15-degree polynomial for $u$ where\n",
    "the coefficients in front of $x^r$ for $r>2$ are\n",
    "of size $10^{-5}$ and smaller. As the matrix is ill-conditioned\n",
    "and we use floating-point arithmetic, we do not obtain the exact\n",
    "solution. Still, we get  a solution that is visually identical to the\n",
    "exact solution. The reason is that the ill-conditioning causes\n",
    "the system to have many solutions very close to the true solution.\n",
    "While we are lucky for `N=15` and obtain a solution that is\n",
    "visually identical to the true solution, ill-conditioning may also\n",
    "result in completely wrong results. As we continue with higher values,  `N=20` reveals that the\n",
    "procedure is starting to fall apart as the approximate solution is around 0.9 at $x=1.0$,\n",
    "where it should have\n",
    "been $1.0$. At `N=30` the approximate solution is around $5\\cdot10^8 $ at $x=1$.\n",
    "\n",
    "\n",
    "### Successful example\n",
    "\n",
    "Trying out the Lagrange polynomial basis for approximating\n",
    "$f(x)=\\sin 2\\pi x$ on $\\Omega =[0,1]$ with the least squares\n",
    "and the interpolation techniques can be done by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3RUxd/H8ffsbnqv9N57C70EhSCoVBFBQUAREBAQpIhSREBAQhOliYCKFCkCCiI1oQgS/NF7J5T0hPS28/yRwBMwlJBN7iaZ1zl7snvbfADdb+69c2eElBJFURSl4NJpHUBRFEXRlioEiqIoBZwqBIqiKAWcKgSKoigFnCoEiqIoBZxB6wAvwt3dXZYuXVrrGIqiKHnKsWPHQqWUHo8vz5OFoHTp0gQEBGgdQ1EUJU8RQtzIbLm6NKQoilLAqUKgKIpSwKlCoCiKUsDlyXsEiqLkruTkZAIDA0lISNA6ivIcrK2tKV68OBYWFs+1vSoEiqI8U2BgIA4ODpQuXRohhNZxlKeQUhIWFkZgYCBlypR5rn1McmlICPGDECJYCHH6CeuFEGK+EOKyEOKkEKJuhnW9hRCX0l+9TZFHURTTSkhIwM3NTRWBPEAIgZubW5bO3kx1j2AF0PYp69sBFdJf/YGFAEIIV2Ai0BBoAEwUQriYKJOiKCakikDekdV/K5NcGpJS+gshSj9lk47AjzJtzOvDQghnIUQRoCWwU0oZDiCE2ElaQVltilxK1qSG3CYxYC8pNy+TfPc2MjEBmZqKzsoavbsHFsVKY1XPG32pygid6megKPlFbt0jKAbcyvA5MH3Zk5b/hxCiP2lnE5QsWTJnUhYwxugIYrcsJ+av7cRdvE1SxPPMTTEfvbXErrw7do0bYv/WQAzFK+R4VkUxF1u2bOHs2bOMHTs228eaO3cu/fv3x9bWFoBXX32VX375BWdn52wfOytyqxBkdp4in7L8vwulXAIsAfDy8lKz6WRDgv9vRCz/jvsBNzEmC3QGiW0ZR5yaV8CqVj0sylbFomwVhIMrQm+BMTqc1DvXSL58msTT/5Jw7jyxF4K4f3obfP8H9uUdce76BvbvjEQYVP8DJe9ISUnBkMX/Zjt06ECHDh1M0v7cuXPp2bPnw0Kwbds2kxw3y6SUJnkBpYHTT1i3GOiR4fMFoAjQA1j8pO2e9KpXr55Usi52yw/y+it15NlKleW5qpXk7e7eMnrNfJkaez/LxzKmpsr4/Vtk0OCO8mKdSvJspcrycsMqMmLWCGlMiM+B9IqWzp49q3UEKaWUHTt2lHXr1pVVq1aVixcvllJKaWdnJ0eMGCHr1KkjX375ZRkcHCyllNLb21sOGzZMNm7cWFarVk0eOXJESinlxIkT5QcffCB9fHxkjx49ZHx8vOzTp4+sXr26rF27ttyzZ4+UUkpfX1/Zt29fKaWUJ0+elNWqVZOxsbFy+fLlcvDgwVJKKXv37i0HDhwoW7ZsKcuUKSP37dsn+/btKytXrix79+79MPfAgQNlvXr1ZNWqVeWECROklFLOmzdPWlhYyOrVq8uWLVtKKaUsVaqUDAkJedh+tWrVZLVq1eScOXOklFJeu3ZNVq5cWfbr109WrVpV+vj4yLi4uEz/rjL7NwMCZCbfqbn169sWYIgQYg1pN4ajpJR3hRA7gGkZbhC3AT7NqRDB0Qk4WFlgY6nPqSbMUmLAHoImjib2Six6G4nnm01wHjIRfaFSL3xModNh3aw91s3a45GYwP3l0whbtZG7S7cRtv5PPAe/h/3bH6t7CfnQF1vPcPbOfZMes2pRRya2r/bM7X744QdcXV2Jj4+nfv36vPHGG8TGxlK3bl18fX2ZPHkyX3zxBQsWLAAgNjaWQ4cO4e/vz3vvvcfp02kdG48dO8aBAwewsbHB19cXgFOnTnH+/HnatGnDxYsXGT58OC1btmTTpk1MnTqVxYsXP/zNPaOIiAj27NnDli1baN++PQcPHuT777+nfv36HD9+nNq1azN16lRcXV1JTU2lVatWnDx5kqFDhzJ79mz27t2Lu7v7I8c8duwYy5cv58iRI0gpadiwId7e3ri4uHDp0iVWr17N0qVL6datGxs2bKBnz57Z+vs3VffR1cDfQCUhRKAQ4n0hxEAhxMD0TbYBV4HLwFJgEIBMu0n8JXA0/TU5fVmOGPXrSdrM9WPvheCcasKsGKMjCB7Siau9BhF/MwbPbk0o7/83bl/+kK0i8DhhZY3TwMmU8TtJ8bG9wSgJnPI9gZ0ak3zxuMnaUZT58+dTq1YtGjVqxK1bt7h06RI6nY633noLgJ49e3LgwIGH2/fo0QOAFi1acP/+fSIjI4G0yzs2NjYAHDhwgF69egFQuXJlSpUqxcWLF9HpdKxYsYJevXrh7e1N06ZNM83Uvn17hBDUqFGDQoUKUaNGDXQ6HdWqVeP69esArFu3jrp161KnTh3OnDnD2bNnn/rnPHDgAJ07d8bOzg57e3u6dOnC/v37AShTpgy1a9cGoF69eg/byA5T9Rrq8Yz1Ehj8hHU/AD+YIsezDPQux2e/naLv8qO8VqMIE9pXpZCjdW40nevid67hzoTJJEVInLyK4jltEYaSFXO0TaHT4dBnLPbdhxI+7SNCNhzk6hvdKdT3dZyGz1RnB/nE8/zmnhP27dvHrl27+Pvvv7G1taVly5aZ9pXP2HXy8W6UDz7b2dk9XJb29ZS5S5cuYW9vz507d564jZWVFQA6ne7h+wefU1JSuHbtGrNmzeLo0aO4uLjQp0+fZ/bxf1qmjG3o9Xri4+OfeqznUaD+z2xczo3tw5oz0qciO88F0crXjxUHr5FqzD/3nqXRSNiE97k+dBLGJEmJKUMo+vOeHC8CGQlrW9wmL6Ps6mVYF7Hh7pI/uNPdm9TQJ//PpCjPEhUVhYuLC7a2tpw/f57Dhw8DYDQaWb9+PQC//PILzZo1e7jP2rVrgbTfsJ2cnHBycvrPcVu0aMGqVasAuHjxIjdv3qRSpUpERUUxbNgw/P39CQsLe9hGVt2/fx87OzucnJwICgpi+/btD9c5ODgQHR2daabffvuNuLg4YmNj2bRpE82bN3+h9p9HgeviYWXQ81GrCrSvVZTxm08zaetZNv7vNtM616B6sf/+R5KXpIbf4+4HbxB9JhyHKi4UWbIevUemvXFzhWXNppT84whh498jZHMA8a+1psS3C7DyelmzTEre1bZtWxYtWkTNmjWpVKkSjRo1AtJ+uz9z5gz16tXDycnp4Zc/gIuLC02aNOH+/fv88EPmFx4GDRrEwIEDqVGjBgaDgRUrVmBlZcWHH37IoEGDqFixIsuWLeOll16iRYsWWc5dq1Yt6tSpQ7Vq1Shbtuwjl5j69+9Pu3btKFKkCHv37n24vG7duvTp04cGDRoA0K9fP+rUqWOSy0CZEU87BTFXXl5e0hQT00gp2XryLpO3niU8NpHeTUozsk0l7K3yXn1MOnOYW/3eJykyFc+ujXH9YplZXYqJ+30FgZ9PR6ZC8YnDsOv6odaRlCw4d+4cVapU0TpGpuzt7YmJifnP8pYtWzJr1iy8vLw0SKW9zP7NhBDHpJT/+Qsxn28KDQgh6FCrKLtHevN2w5KsOHSd1r5+bD9196nX6MxNgv9vXO/Zh5TYVEp+NRK3L5ebVREAsH29D6V/WYmFo56b4+cRMWOo1pEURUlnXt8WGnGysWBKpxps/LAJLnaWfLjqX95fGcCt8Ditoz1T7PqF3Bg8FqEXlF62ALtOH2gd6Yksqzak1Oad2JVz4N7ynYSOzV6XN0UBMj0bgLSbywX1bCCrVCHIoE5JF7YOacpnr1bh8NUwfOb4scjvCsmpRq2jZSp65QxuTpiHhaOB0mvWYFW/tdaRnknvXpQSG/bjWMuDkN+OETy0C9Jonn+/ilJQqELwGINexwctyrJzhDfNK3gwfft5Xp9/gGM3cuzxhhcS/dMsAmcsx9rTklKbtmNRvpbWkZ6bsLKm6Ko9ODcsTthf5wga2F4VA0XRkCoET1DM2Yal73qxpFc9ohOSeWPh33y68SSRcUlaRyPmlznc/up7rD0sKLnuD/SeJbSOlGXCYKDw8h24epcjwv8qIcO6ah1JUQosVQieoU21wuwc4c0HzcuwLiCQVr5+bPw3ULObybEbFxM4ZTFW7haUXPd7niwCDwidDs+FW3BuXIKwnecIHfeu1pEUpUBSheA52FkZ+Oy1qmwZ0pQSrraMWHeCd74/wpWQzG9S5ZSEA1sJnDgHS2c9JdduNekwEVoROh2Fl27DsZY7IRuPEjH9I60jKQXQvn37OHTokNYxNKMKQRZUK+rEhg+b8GWn6py6HUW7ufuZs/MiCcmpOd528rmj3Bo6Cp2loMSKVeiLlM7xNnOLMBgounIH9hUcuLdiJ9ErZ2gdSSlgVCFQskSvE/RqVIrdI71pW70w83Zfot28/Ry8HJpjbabevc7N93pjTIGS383FomLtHGtLK8LalmI/b8e6sCW3Zy4nft8mrSMpZujnn3+mQYMG1K5dmwEDBnDjxg0qVKhAaGgoRqOR5s2b89dffwHQqVMn6tWrR7Vq1ViyZMnDY/z555/UrVuXWrVq0apVK65fv86iRYuYM2cOtWvXfji4W0FSoJ8sNgX/iyGM33yaG2FxdKpdlM9eq4qHg9Wzd3xOMjGBWx2bEnczlpIzxmDbvq/Jjm2OUq6d5dqbb4BRUnrNmnxZ9PKiR55S3T4W7p0ybQOFa0C76c/MMHr0aDZu3IiFhQWDBg2iUaNGJCUl8eeff9KwYUMuX77M4sWLAQgPD39kyGo/Pz+MRiN169bF39+fMmXKPNxm0qRJ2Nvb88knn5j2z6Uh9WRxLmpR0YMdw1sw9OXy/HHqLq189/HLkZsYTTSQXfDQrsRej6PwB+3zfREAMJSpSon5vhiT4db7PTFG5tyZlpK37N69m2PHjlG/fn1q167N7t27uXr1Kv369SM6OppFixYxa9ash9tnNmT14cOHadGiBWXKlAHA1dVVqz+OWcl7g+qYIWsLPSPaVKJD7WJ8tukU4zadYv2xW0zrUoPKhR1f+LiRc0cT7ncFlxZlcf74axMmNm/WTV6l2LhL3PpiIXcHdKboaj+zGzKjQHvGb+45RUpJ7969+eqrrx5ZHhcXR2BgIJD2lLGDg8MTh6yWUv5naGpFnRGYVHlPe9b0b8SsN2txPSyO1+Yf4Ktt54hLSsnyseJ3rePeki3Ylbal0DcbciCtebPvMQyPjvW4fyKUiGmZTmWhFDCtWrVi/fr1BAenTSwVHh7OjRs3GDNmDO+88w6TJ0/mgw/Shlh50pDVjRs3xs/Pj2vXrj08Bjx5OOiCwlQzlLUVQlwQQlwWQozNZP0cIcTx9NdFIURkhnWpGdZtMUUeLQkh6FqvOLtHeNO1bnEW+1/FZ7Y/u88FPfcxUoNucHvsRAy2gmLLNyCs8ufkOc/iNnUl9pUcCVq1l7g/VmodR9FY1apVmTJlCm3atKFmzZr4+Phw/fp1jh49+rAYWFpasnz5ctq2bUtKSgo1a9Zk/PjxD4es9vDwYMmSJXTp0oVatWo9nNmsffv2bNq0Sd0sfuEDCKEHLgI+QCBpU072kFJmOhebEOIjoI6U8r30zzFSSvustGlON4uf5Z9r4Xy26RSXgmN4pVohJnWoRhEnmyduL41GAjs3IeZSJKXnT8amdbdcTGt+UoNvca39KxiTjJT97TcMpSprHalAMudhqJXM5fbN4gbAZSnlVSllErAG6PiU7XsAq03Qbp7QoIwrfwxtzqhXKrHvQgitff1YduAaKU8YyC7iy4HEXIjCs1vzAl8EAPSeJSg+ZxbGJLgzqBcyJeuX2RRFeTpTFIJiwK0MnwPTl/2HEKIUUAbYk2GxtRAiQAhxWAjR6UmNCCH6p28XEBISYoLYucfSoGPwS+XZ+bE39cu48uXvZ+n47UFO3Ip8ZLv4XesIWuuPfSVHXCcs1iit+bFu8iqe77Qi9koMEVMHaR1HUfIdUxSCzG7BP+l6U3dgvZQy46O4JdNPVd4G5gohymW2o5RyiZTSS0rp5eHhkb3EGinpZsvyPvX59u26hEQn0um7g0zYfJr7Ccmkhtzm9qcTsbAVFF20VvWSeYzL2G+wr+RI8Fp/Evx/0zpOgZQXnzkqqLL6b2WKb5tAIOPIZ8WBJ81S3p3HLgtJKe+k/7wK7APqmCCT2RJC8FrNIuwa6c27jUrx0+EbtPb14+qH3UmOkRSbOiFfDR9hKkKno8i3v6C3htujx6nnC3KZtbU1YWFhqhjkAVJKwsLCsLZ+/k4mprhZbCDtZnEr4DZpN4vfllKeeWy7SsAOoIxMb1QI4QLESSkThRDuwN9AxyfdaH4gL90sfpaTgZEcmz6ORrv2Elq3FKW+2UhJN1utY5mt2I2LuTluDs4NilHkx91axykwkpOTCQwMJCEhQesoynOwtramePHiWFhYPLL8STeLs/1AmZQyRQgxhLQveT3wg5TyjBBiMhAgpXzQJbQHsEY+WnmqAIuFEEbSzk6mP6sI5DdVjcFYH9iDcNczvMxAYub4MbRVBT5oXhZLg7o89Di7LgNw2/MnYbvO4/DLHOzf/ljrSAWChYXFw6dxlfxHjTWkoQddRWMvRVJm+TwiKnvzxdYzbD99jwqe9kztXIMGZdQj8I8zxkVzvXUjUhNSKfvnzjw9J4Oi5CY11pAZipo/lpgLUXi82Qyrhq9Q2MmahT3r8UMfL+KSUum2+G9G/XqC8FjtZ0UzJzpbB4pM+YKUeAj6WE1moyjZpQqBRpIvnyBo2RZsS1jjOn7RI+terlyInSNaMMC7LJv+d5tWvvv4NeCWulGXgc3LXXF7pTpRx+4RvWq21nEUJU9ThUAD0mjk3sj+SCMU8V2AMPz3Vo2tpYFP21Xh96HNKOthz6j1J3lryWEuBxfc8VAe5z5tOVZuOu75LiE16IbWcRQlz1KFQAPRP0wl5sJ9PLo0wbJm06duW7mwI78OaMxXXWpw4V407ebtZ9aOC7kyK5q509k6UGTq5LRLRCP6aB1HUfIsVQhyWWrQDe59uwprTwOun3/3XPvodIIeDUqye6Q37WsWZcHey7wy1x//i3nrCeucYNPyDdx8qhJ17B6xm7/XOo6i5EmqEOSyoE/eIzUBikz5MsujirrbWzH7rdr80q8heiF494d/GPLLvwTfL9h9u92nLsPCEe59NRtjnLp0pihZpQpBLorduJioo3dwa1UZ6xZPHFbpmZqUd2fbsOYMb12Bv84E0crXj5/+vk6qiWZFy2t0Di4U/mQwSZGSsAkfaB1HUfIcVQhyiTEumnsz5mHhmPYbbHZZW+gZ3roifw5vTs0STozffIYuCw9x5k6UCdLmPfbdhuBY052wbcdJ/NdP6ziKkqeoQpBLwid/SFKUpPCID9E5uZnsuGU97Pn5/YbMfas2tyPiaP/NAb78/SyxiQVvuOZCMxYjDHBv7HCkMfNhvhVF+S9VCHJB8vljhG4NwKGyE/bdh5r8+EIIOtUpxu4RLeneoCTLDlyj9Ww/dpy5Z/K2zJmhTFU8e75C3M0EohaM0zqOouQZqhDkgqBxHwFQ6KtvcrQdJ1sLpnWuwYYPm+BkY8GAn47Rb2UAgRFxOdquOXEe4YtNEQuCl/9GauiTBsFVFCUjVQhyWMy6BUSfjcD99XpYVKmfK23WK+XC1o+a8Wm7yhy8HIrPbH+W+F8h+QmzouUnwmCg0OefkxoPoRMGah1HUfIEVQhykDEumqDZ32HhJHCduOjZO5iQhV7HAO9y7BzRgibl3Ji27TztvznAvzcjcjWHFmxadcOpfjHC914kMWDPs3dQlAJOFYIcFD4lrUtj4Y8HorN10CRDcRdbvu/txaKe9YiMS+aNhYf4bNMpouKSNcmTWzynLkRngKAJo9WNY0V5BlUIckjyxX8J3fwP9jl0gzgrhBC0rV6YXSO96dukDKv/uUmr2fvYfPx2vh3IzlCyIh5vehN7NZaYn2dpHUdRzJoqBDkkePxwAApNm6dxkv9nb2VgQvuqbBnSjKLONgxbc5xey/7hWmis1tFyhMvoOVi66ghasBxjTMF8vkJRnodJCoEQoq0Q4oIQ4rIQYmwm6/sIIUKEEMfTX/0yrOsthLiU/uptijxai9+1jvsnQnBtXQ3Lqg21jvMf1Ys5sWlQUyZ3rMaJW5G8MtefebsukZiSvwayE9a2FB4xhOT7aZfpFEXJnCnmLNaTNmexD2kT2R8FemScclII0QfwklIOeWxfVyAA8AIkcAyoJ6V86h1Nc56hTBqN3Ghdh6SIRMrt3ofetbDWkZ4q6H4CX/5+lt9P3qWshx1TOlWnSTl3rWOZVGDnxsRcjKDc5rVYlK+ldRxF0UxOzlDWALgspbwqpUwC1gAdn3PfV4CdUsrw9C//nUBbE2TSTPT3XxJ/JwmPd141+yIAUMjRmgVv12VF3/okpxp5e+kRRqw7TlhMotbRTMZzyhyQEDJJzW+sKJkxRSEoBtzK8Dkwfdnj3hBCnBRCrBdCPJhk9nn3RQjRXwgRIIQICAkxz+GXjTFRBC9dg5W7Hudh07WOkyUtK3ny13BvBr9Ujq0n7vCyrx9r/rmJMR8MZGdZrREuLSsSFXCHhIN/aB1HUcyOKQqByGTZ498eW4HSUsqawC5gZRb2TVso5RIppZeU0svDw+OFw+ak8K+GkhwNhUYMQVhYah0ny2ws9Yx6pTLbhjanUiEHxm48RbfFf3PhXt4f2tl94jforSB4ygTVnVRRHmOKQhAIlMjwuTjwyLP9UsowKeWDaw1LgXrPu29ekXLzImFbjmBf0RG7Lnn7idYKhRxYO6ARM7vW5EpIDK/N38/07eeJT8q7N5P1hUrh/oY3sdfiiF23QOs4imJWTFEIjgIVhBBlhBCWQHdgS8YNhBBFMnzsAJxLf78DaCOEcBFCuABt0pflOSETh2BMAc9JeeuS0JMIIejmVYLdI1vSqU4xFvldwWeOH3vPB2sd7YW5jJqNhSMEz1+MTCzYk/koSkbZLgRSyhRgCGlf4OeAdVLKM0KIyUKIDumbDRVCnBFCnACGAn3S9w0HviStmBwFJqcvy1MSA/YQefgmLk3LYlX3Ja3jmJSrnSWz3qzFmv6NsDLo6LviKINWHeNeVN77IhU2dngOeIfEcCOR36jRSRXlgWx3H9WCuXUfDezcmNjLEZTbvhVD8Qpax8kxiSmpLPW/yjd7LmOh1zGyTUXebVwavS6zWz3mKWP33vJ7/NG5eGodSVFyTU52Hy3Q4neuIfpcJK5t6+TrIgBgZdAz5OUK/PVxC+qUdOaLrWfp9O1BTgXmnad2hU5HoTFjSY0XhE35SOs4imIWVCHIBmk0EjxzBnprids48xlKIqeVcrPjx/ca8E2POty7n0DHbw8wacsZohPyxkB2Nq/0wKGaC+E7TpBy86LWcRRFc6oQZEPs2m+Iu5WAe7dWBe4SgxCC9rWKsmuEN+80LMXKv6/TerYf207dzRMD2Xl8NhVjKoROUQ+ZKYoqBC9IpqQQ/O1SLBzB5eOZWsfRjJONBV92qs7GD5vgZmfFoFX/8t6Ko9wKN+9Z0azqvoSTVzEiD1wh+dxRreMoiqZUIXhB9xdPIjE0FY++byJs7LSOo7k6JV3YMqQpn79WhSPXwvGZ48d3+y6b9axoHuO/BiBkymiNkyiKtlQheAHGuGhCVmzAykOP4wcTtI5jNgx6Hf2al2XXCG9aVPBg5p8XeG3+fgKum2ePYIuKdXFpUYGof++S+M9OreMoimZUIXgBkb6jSI4Gz48GIgwGreOYnaLONix514ul73oRm5hK10V/M3bDSSLjkrSO9h9un89FZ4CQr1RBVwouVQiyKDX8HqEb9mFbyga7roO0jmPWfKoW4q+PW9C/RVl+PRbIy75+bDgWaFY3kw3Fy+Hatg7R5yKJ37VO6ziKoglVCLIoYvpIUhMEnqPGInTqr+9Z7KwMjHu1CluHNKOUmy0jfz3B20uPcCUkRutoD7l+Oge9tSR4Vv4YHkRRskp9k2VBatANwrYfw76SEzatu2kdJ0+pWtSRDQObMLVzdc7ciaLd3P3M/usCCcnaD2Sndy2Mexdv4q7HE7t+odZxFCXXqUKQBWHTP8GYLPAYPV7rKHmSTid4p2Epdo9sSbsahZm/5zJt5/pz4FKo1tFw/mQWBnsI/uY7NUy1UuCoQvCcUgKvELHzFA7VXLFu+prWcfI0Dwcr5nWvw0/vNwCg57IjDFvzP0KitZsVTWfrgMfbr5EQlELMz7M0y6EoWlCF4DmFfTUCYyp4jP1C6yj5RvMKHvw5vAVDW1Vg+6l7vOy7j58P39BsVjSnwVOwcISQJT8iU1I0yaAoWlCF4DmkXDtLxL4LONYqhFX91lrHyVesLfSM8KnItmHNqVbUkc9/O80biw5x9s79XM8irKzx6NWJxNBUon8suE+LKwWPKgTPIXTaJ0gjeHw6Reso+VZ5T3tWf9CI2d1qcSMsjvYLDjD1j7PEJubub+aO/Sdi6SwIXbZKnRUoBYZJCoEQoq0Q4oIQ4rIQYmwm60cIIc6mT16/WwhRKsO6VCHE8fTXlsf31VryxeNEHryKk1cxLGs11zpOviaEoEvd4uwZ6c2b9YqzdP812szxZ+fZoNzLYGWNe++uJIYZif5haq61qyhayvbENEIIPXAR8CFtDuKjQA8p5dkM27wEHJFSxgkhPgRaSinfSl8XI6W0z0qbuTkxzd2+bYg8cpPyG37Cokr9XGlTSXP0ejifbTrFxaAY2lQtxKQO1SjqbJPj7crkJK561wEJZf3/h7CwzPE2FSU35OTENA2Ay1LKq1LKJGAN0DHjBlLKvVLKB8NRHiZtknqzl3TmcNoUlI1KqSKggfqlXfn9o+aMaVsZ/0shtJ7tx/f7r5KSwwPZCQtLPPq+RVKEkftLv8zRthTFHJiiEBQDbmX4HJi+7EneB7Zn+GwthAgQQhwWQnR60k5CiP7p2wWEhIRkL/FzCp02DiHAbZy6cagVS4OOD1uWY+fH3jQs48qUP87RYcFBjt+KzNF2HfqOw8pNT+iPG9RE90q+Z4pCkNmEtZlebxJC9AS8gK8zLC6ZfqryNhB8j8UAACAASURBVDBXCFEus32llEuklF5SSi8PD4/sZn6mxH/9iPr3Di7Ny2NRvlaOt6c8XQlXW37oU5/v3qlLWGwinb87yPjfTnM/h2ZFEwYD7v3eISlSErV4Uo60oSjmwhSFIBAokeFzceDO4xsJIVoDnwEdpJQPnxySUt5J/3kV2AfUMUGmbAv7eiJCB27j1MNF5kIIwas1irBrhDe9G5dm1ZEbtPL1Y8uJOzkykJ1D7zFYeegJ/XkzMsG8J9pRlOwwRSE4ClQQQpQRQlgC3YFHev8IIeoAi0krAsEZlrsIIazS37sDTYGzaCzpxH6ijt/DpXkFDKUqax1HeYyDtQWTOlTjt8FNKexozdDV/+PdH/7hRlisSdsROh0eH7xL8n2I+k4NU63kX9kuBFLKFGAIsAM4B6yTUp4RQkwWQnRI3+xrwB749bFuolWAACHECWAvMD1jbyOthM5MPxsYM0PrKMpT1CzuzG+DmzKxfVX+dzOSNnP8+Wb3JRJTTDeQnX3PT7D2NBC6+g9kvGkLjaKYi2x3H9VCTnYfTTp1iCvd3sOleTkKL/kjR9pQTO9eVAKTfz/DtlP3KOdhx9TONWhU1s0kx475ZQ63Ji+h8HttcBk9zyTHVBQt5GT30XwldOb4tJ5CY9TY9HlJYSdrvnunHsv71CcxxUj3JYf55NcThMdmf1Y0u+7DsC5iQejav9RZgZIvqUKQQdLZI0QF3Ma5SRksytXQOo7yAl6q7MnOj70Z6F2O3/53m5d997Hu6K1s3UwWOh0eA/qREguRCz4zYVpFMQ+qEGQQNuPztLOB0V9pHUXJBhtLPWPbVeaPoc0p72HP6A0neWvxYS4FRb/wMe26DcG6kAVh63aoHkRKvqMKQbrk88eIPHoLp4alsKhYW+s4iglUKuzAugGNmd6lBheConl1/n6+3nH+hWZFEzod7h/0JjkaohZOzIG0iqIdVQjShc4YB4D7mGkaJ1FMSacTdG9Qkj0jvWlfqyjf7r1Cmzn+7LsQ/OydH2P/9sdpzxWs/l09bazkK6oQkDbCaNSRGzjXL4FF5Xpax1FygJu9FbO71eaXDxpi0An6LD/K4F/+Jfj+83+hC50Oj/d7knwf7i9RExQp+YcqBEDYzE+REtzGqPkG8rsm5dzZPrw5H7euyM6zQbTy9ePHv6+T+pyzotn3/AQrdz2hqzYjk7PfI0lRzEGBLwTJV04ReegazvWLYVm1odZxlFxgZdAzrHUFdgxvQa0SzkzYfIYu3x3k9O2oZ+4rDAbc+3YnKVJy/3v1i4OSPxT4QhA2Y2za2cAoNdxwQVPG3Y6f3m/AvO61uR0ZT4cFB5i89Swxz5gVzaH3WKxcdWkjk6pZzJR8oEAXguRrZ4g8eAWnukWxrNFE6ziKBoQQdKxdjN0jWtK9QUl+OHgNn9l+/Hn63hOfPRAGA+693yQpwkj0ctW5QMn7CnQhCJ8+BmkE9zHqbKCgc7K1YFrnGmz4sAlONhYM/PkY/VYGEBiR+TMDDu+Nw9JFELpinTorUPK8AlsIUm6cJ+LAZZxqF8ayZlOt4yhmol4pF7Z+1Ixxr1bm0JUwfGb7s9jvCsmPzYomLCxx79mFxLBUon/6+glHU5S8ocAWgrAZo5FGcBulugEqj7LQ6+jfohy7RnrTtLw7X20/T/tvDnDsRsQj2zn2+xxLJ0HoD6uQxpydPlNRclKBLAQpNy8S4XcRx5qeWNX11jqOYqaKOdvwfW8vFveqR1R8Mm8sPMSnG08RFZc2K5qwssbt7fYkhqQS87OawEjJuwpkIQifOQaZCu6jJmkdRckDXqlWmJ0jvHm/WRnWHr1Jq9n7+O1/t5FS4jTwCywcIfT7n9RZgZJnFbhCkBJ4hfB953Cs6YGV18tax1HyCHsrA+Nfr8qWIc0o5mLL8LXH6bnsCNfup+De/TUSglOIXaPmKlDyJpMUAiFEWyHEBSHEZSHE2EzWWwkh1qavPyKEKJ1h3afpyy8IIV4xRZ6nCf96NDIF3D9RUw8qWVe9mBMbP2zClx2rcfJWFG3n7uenKu9isIeQpcvVWYGSJ2W7EAgh9MC3QDugKtBDCFH1sc3eByKklOWBOcCM9H2rkjbHcTWgLfBd+vFyROrd60TsOYNDdTesGvjkVDNKPqfXCXo1Ls3ukd68Ur0wvn63+F+N2iTcTSZ2/Xdax1OULDOY4BgNgMtSyqsAQog1QEcenYS+IzAp/f16YIEQQqQvXyOlTASuCSEupx/vbxPk+o+tX/SkUrJgno8boX/2zYkmlILGBeo0TGbWvVjmn4Zbi77jR9t/sDDk2O8zSgHlejuC9huv0WT6QpM/AGuKS0PFgFsZPgemL8t0m/TJ7qMAt+fcFwAhRH8hRIAQIiAkJOSFglrFJHG+igWhpUwzl62iADjbWFCltCv7GjtS4o7E8u9rBEcnah1LyWcab7+G241EdC4eJj+2Kc4IRCbLHn82/0nbPM++aQulXAIsgbTJ67MS8IF2P/+DTE6is4Xli+yuKE9lbB7F5QMN+eRAJK+mjsW6lAtTO1encmFHraMpeVzScX+unPkb15cqYiheweTHN8UZQSBQIsPn4sCdJ20jhDAATkD4c+5rUkIVASWH6OydcO/cEhGUws8lznM1JIbX5x/gq+3niEtSw1AoLy7064kIHbiNmp4jxzdFITgKVBBClBFCWJJ283fLY9tsAXqnv+8K7JFpI3ptAbqn9yoqA1QA/jFBJkXRhPPwr9DbSEpsWcXukS3pXKcYi/2u4jPbnz3ng7SOp+RBSWcOE/XvXZyblsNQ5vF+OKaR7UKQfs1/CLADOAesk1KeEUJMFkJ0SN9sGeCWfjN4BDA2fd8zwDrSbiz/CQyWUmZ9QllFMRM6BxfcOjYn7kY81vtW8/WbtVjbvxE2lnreWxHAwJ+OcTcqXuuYSh4SNuNzhAC3MTlzNgAgnjTUrjnz8vKSAQEBWsdQlEwZo8K43LIp1oXsKPnnMQCSUows3X+V+bsvYdAJRrapxLuNS2HQF7hnOpUsSD5/jMtd3sGlcWkKL/sz28cTQhyTUno9vlz9V6goJqZzcsOtfVNir8cRt/1nACwNOga/VJ6/Pm5BvdKuTP79LJ2+O8iJW5Eap1XMWdjMcQC4jcnZeS9UIVCUHOAyYgZ6a0noN3MfWV7KzY6Vfeuz4O06BN1PpNN3B5m4+TT3E5I1SqqYq+TLJ4g8fAPn+iWwqFg3R9tShUBRcoDO2R239o2JvRpL/I7Vj6wTQvB6zaLsHulNr0al+PHwDVr7+vHHybtPnBVNKXjCZnyaNo3umJyfG1sVAkXJIS4jv0ZvLQmZ75vpekdrCyZ3rM6mQU1xt7di8C//0nfFUW6GZT4rmlJwpFw7S+Shq2nT6FZtmOPtqUKgKDlE5+yO62sNib0SS/zONU/crnYJZ7YMacr416ty9Fo4PnP8+HbvZZJS1AB2BVXYzLFp0+iOzp2Js1QhUJQc5PrJ1+itJCFzMz8reMCg1/F+szLsGunNS5U8+XrHBV6bv5+j18NzKaliLlJuXiTC/yJOtQthWat5rrSpCoGi5CCdiyeur9Yn9koM8bvWPXP7Ik42LOpVj2W9vYhLSuXNRX8zZv1JImKTciGtYg7Cpn+SPo3u5FxrUxUCRclhLqPSzgpC5z3/dJatqhRi54gWDGhRlvX/BtJqth/rjwWqm8n53INpdJ1qF8rVaXRVIVCUHKZ3LYxr23rEXIomfs/6597P1tLAp69W4fePmlHazZZPfj1B9yWHuRwck4NpFS1pcTYAqhAoSq5wGfU1OktJ6Nyvs7xvlSKOrB/YhGmda3Du7n3azfPH968LJCSr0VjyE63OBkAVAkXJFXr3ori2rUvMxfvE79uU5f11OsHbDUuye2RLXqtRhG/2XOaVuf7sv/Ric3Mo5kerswFQhUBRco3rg7OCOTNe+BgeDlbM7V6Hn99viE4Iei37h6Gr/0dwdIIJkyq5TcuzAVCFQFFyjd6jGK5tahNzIYqE/Y+P1J41zSq4s31Yc4a1qsCfp+/RytePnw7fwGhUN5PzIi3PBkAVAkXJVa6jv0ZnIQn1zf4gYtYWej72qcj24c2pUcyJ8b+dpsvCQ5y5E2WCpEpueXA24FhLm7MBUIVAUXKV3rMErj61iD4fRcLBP0xyzHIe9qzq15A5b9XiVngcHRYcZMrvZ4lNVLOi5QVhM0alP0U8SbMMqhAoSi5zHTMz/axgqsmOKYSgc53i7B7pTTev4nx/4Bo+s/3468w9k7WhmF7KzYtE7LuQfjbwkmY5slUIhBCuQoidQohL6T9dMtmmthDibyHEGSHESSHEWxnWrRBCXBNCHE9/1c5OHkXJC/SFSuHSugbRZyNIOLTNpMd2trXkqy41WT+wMQ7WFvT/6Rgf/BjA7Ug1K5o5MoezAcj+GcFYYLeUsgKwO/3z4+KAd6WU1YC2wFwhhHOG9aOklLXTX8ezmUdR8gS3h/cKTHdWkJFXaVd+H9qMse0qs/9SCD6z/Vjqf5WUVDWQnbkwl7MByH4h6AisTH+/Euj0+AZSyotSykvp7+8AwYBHNttVlDxNX6Q0Li9XI/pMGAl/Z38KwsxY6HUM9C7Hzo+9aVTWjanbztF+wUH+vRmRI+0pWWMuZwOQ/UJQSEp5FyD9p+fTNhZCNAAsgSsZFk9Nv2Q0Rwhh9ZR9+wshAoQQASEh6iEaJe9zG+uLzgJCZ+bsUMMlXG1Z1tuLhe/UJTw2kTcWHuKzTaeIilezomnFnM4G4DkKgRBilxDidCavjllpSAhRBPgJ6CulfHB++ilQGagPuAJjnrS/lHKJlNJLSunl4aFOKJS8T1+kNK6v1Cb6XCTx+zbkaFtCCNrVKMKuEd70aVKa1f/cpJWvH5uP31YD2WkgbNpIszkbgOcoBFLK1lLK6pm8NgNB6V/wD77ogzM7hhDCEfgD+FxKeTjDse/KNInAcqCBKf5QipJXuI7xTZuvYNb0XGnPwdqCie2rsXlwM4o6WzNszXHe/eEfrofG5kr7StpcxBH+l3CqW8QszgYg+5eGtgC909/3BjY/voEQwhLYBPwopfz1sXUPiogg7f7C6WzmUZQ8Re9RDNfXGxJ7OYa47T/nWrs1ijuxaVBTvuhQjf/djKTNXH/m775EYooayC6nhU4dhQTcP8ud4v88slsIpgM+QohLgE/6Z4QQXkKI79O36Qa0APpk0k10lRDiFHAKcAdyfpZmRTEzrqN80+Y2njs7V9vV6wS9m5Rm90hvfKoUYvbOi7Sbt5+/r4Tlao6CJOnMYSIP38S5YalcmYv4eYm8eH3Qy8tLBgQEaB1DUUwmfPIAgn7xp+S04dh1GaBJhr0Xghn/22kCI+J5o25xxr1aGTf7J/bfUF7Anbdbcv/EPcpt+RWLcjVyvX0hxDEppdfjy9WTxYpiBpxHzMRgByELvkMatenr/1IlT3Z+7M2gluXYfPw2rWb7sfboTTWQnYkkHt1F1P/u4dKioiZF4GlUIVAUM6Czd8K9mw/xd5KIXfuNZjlsLPWMbluZbcOaU8HTnjEbTvHWkr+5GBStWab8ImTGRHR6cPvMV+so/6EKgaKYCeehX2HhACELv9fsrOCBioUcWNu/MTPfqMml4BhenbefGX+eJz5J3Ux+EQn7txB9OhwXnxoYilfQOs5/qEKgKGZC2Njh3rMDCcEpRK988clrTEWnE3SrX4LdI7zpWLsYC/ddoc1cP/ZeyLSXuPIUITOnoLOUuH2aux0CnpcqBIpiRpwGfoGlsyB06c/I5CSt4wDgZm+Fb7darP6gERZ6HX2XH2Xwqn8Juq9mRXse8TtWE3MpGrfX6qP3LKF1nEypQqAoZkRYWePRtxuJ4UbuL56kdZxHNC7nxvZhzRnpU5Gd54Jo5evHioPXSFU3k58qeM7X6K0lrqPN797AA6oQKIqZcXj/c6w9DYSs3IQxzrxu0loZ9HzUqgJ/DW9BnZLOTNp6ls7fHeT0bTUrWmZiNy4m7no87p1boHN56lBsmlKFQFHMjDAY8Bw6iORoiPx6pNZxMlXa3Y4f32vAvO61uROZQIcFB/hi6xli1KxoD8mUFILnLcBgB84jv9Y6zlOpQqAoZsiu64fYlbEldKM/qSG3tY6TKSEEHWsXY/dIb95uWJIVh67T2teP7afuqoHsgOhlU0gISsGjd0d09k5ax3kqVQgUxUx5jPmc1ERB2NRhWkd5KicbC6Z0qsGGD5vgYmfJh6v+5f2VAdwKj9M6mmZkfCzBy9Zi5a7H6cPJWsd5JlUIFMVM2bTsjGNNd8J3nib52hmt4zxT3ZIubB3SlM9ercLhq2H4zPFj4b4rJBfAWdEiZo0k+T54DumPsLDUOs4zqUKgKGbMY/wMpITQScO1jvJcDHodH7Qoy84R3jSv4MGMP8/z+vwDBFwP1zparkkNuU3ohn3YlrLBrtsQreM8F1UIFMWMWdZogkuTskT+c4vEgD1ax3luxZxtWPquF0t61SM6IZmui/7m040niYwzj2cjclLYlKGkJgg8P/0MocsbX7F5I6WiFGDuk+ahM0DI1M+0jpJlbaoVZucIb/o1K8O6gEBa+fqx8d/AfHszOfnyCcJ3ncGxpjs2Ld/QOs5zU4VAUcycoXgFXNvVJfpcJHE7ftE6TpbZWRn4/PWqbBnSlBKutoxYd4J3vj/ClZAYraOZXOjkEUgJHhPNu7vo47JVCIQQrkKInUKIS+k/XZ6wXWqGSWm2ZFheRghxJH3/temzmSmK8hi3T+eit5EEz5ih+YB0L6paUSc2fNiELztV59TtKNrN3c/snRdJSM4fA9kl/rOTyKO3cW1eHstqjbSOkyXZPSMYC+yWUlYAdqd/zky8lLJ2+qtDhuUzgDnp+0cA72czj6LkSzoXTzx7vkb8nSSil5p/d8Qn0esEvRqVYvdIb9pWL8z83ZdoN28/By+Hah0t24K//AydAdzGz9M6SpZltxB0BFamv19J2rzDzyV9nuKXgfUvsr+iFDROQ7/CykNP0NK1GKMjtI6TLZ4O1szvUYcf32uAUUre+f4Iw9f8j5DoRK2jvZCYdQuIuRSNe4dGGIqX0zpOlmW3EBSSUt4FSP/5pME0rIUQAUKIw0KIB1/2bkCklPLBM+mBQLEnNSSE6J9+jICQkJBsxlaUvEdYWFLok+GkxED4lLzRLfFZWlT0YMfwFnz0cnn+OHWXVr77+OVI3poVTSbEETR3IRaO4PKZdpMKZcczC4EQYpcQ4nQmr45ZaKdk+jyZbwNzhRDlAJHJdk/815dSLpFSekkpvTw8PLLQtKLkH3Yd++FQxZnQP46RfOWU1nFMwtpCz8g2ldg+rAVVijgybtMpui46xLm797WO9lwiZo0kKdxIocF90Nk6aB3nhTyzEEgpW0spq2fy2gwECSGKAKT/zHTGCinlnfSfV4F9QB0gFHAWQhjSNysO3Mn2n0hR8jnPL2aBEUImmPfQE1lV3tOeNf0bMevNWlwLjeX1bw7w1bZzxCWZ70B2KXeuEbJuL7albbDvNUrrOC8su5eGtgC909/3BjY/voEQwkUIYZX+3h1oCpyVaR2J9wJdn7a/oiiPsqzZFJeXKhF17A7x+zZoHcekhBB0rVecPSNb8kbdYiz2v4rPbH92nQ3SOlqmQicMwpgMhSZMyTMPj2Umu8mnAz5CiEuAT/pnhBBeQojv07epAgQIIU6Q9sU/XUp5Nn3dGGCEEOIyafcMlmUzj6IUCO6TvkNvDUFTvsyz3UmfxsXOkplda7FuQGNsLfX0+zGAAT8FcDcqXutoDyUe2UHEwWu4NC6FdZNXtY6TLSIvPuHn5eUlAwICtI6hKJqKmDmMez/8RdFhb+aJES5fVFKKkaX7rzJ/9yUMOsGINpXo3bgUBr12v4FLo5FbrzcgPjCGctv+yDM9hYQQx9Lv1z4i757LKEoB5/zx11gXMhC8dB2pofn39pqlQcfgl8qz82Nv6pdx5cvfz9Lx24McvxWpWaaYn2cRezUWj67eeaYIPI0qBIqSRwkLSwpPnEhKHIR+1l/rODmupJsty/vU59u36xISnUjn7w4yYfNp7ick52oOY2QoQfOXY+Wmw2X0nFxtO6eoQqAoeZjNy11xblyScP/LJBzYqnWcHCeE4LWaRdg10pt3G5Xip8M3aO3rx+8n7+TaQHahE/qTHAOFx41GWNvmSps5TRUCRcnjPKctQW8F9yZ8jkwx366WpuRobcEXHauzeXBTPB2tGPLL/+iz/Cg3w3J2VrTEo7sI23UWp3qFsX2t97N3yCNUIVCUPE5fpDSefToQfyeJqG8+1TpOrqpZ3JnfBjVlwutVCbgejs8cP77de5mkFNP3pJJGI/c+G4XOAJ4zlpj8+FpShUBR8gGnIdOwKWZF8MqtpN69rnWcXGXQ63ivWRl2jfTmpUqefL3jAq/O38+Rq2Embef+oonE3UzA820fDMUrmPTYWlOFQFHyAWEwUHjyVFITIXjsB1rH0UQRJxsW9arHst5exCel8taSw4z69QThsdmfFS016AZBS3/FupAFziN9TZDWvKhCoCj5hHXT13BtWZHII4HEbv7+2TvkU62qFGLniBYM8C7Lpv/dppXvPn4NuJWtm8lBI/uSmgCFv/giT0xGn1WqEChKPuIxfTkWjnB32myMUaa9NJKX2Foa+LRdFX4f2oyyHvaMWn+St5Yc5nJwdJaPFbt+IVEBd3FrXRWblp1zIK32VCFQlHxE5+RGkXEjSI6ShIx7T+s4mqtc2JFfBzTmqy41uHAvmnbz9jNrx4XnnhXNGBnK3RnzsXQWuE/LvyPgqEKgKPmMXacPcG5QjPA9F4jfs/7ZO+RzOp2gR4OS7B7pzes1i7Jg72XazPHH7+Kz5zUJGdOH5Ggo8vlodA6ZzsSbL6hCoCj5kOes5Rhs4O74Scj4WK3jmAV3eyvmvFWbVf0aotcJev/wD0N++Zfg+wmZbh+/YzXhfpdxaVoK29f75G7YXKYKgaLkQ3rPEhQe/h6JYamEfq4uEWXUtLw724c1Z3jrCvx1JohWvn789Pd1UjPMimaMjuDOpCkY7AUeX6/QLGtuUYVAUfIph3dH41jbk9BtJ4jfuUbrOGbF2kLP8NYV+XN4c2qWcGL85jN0WXiI07ejAAge8Q5JEUaKjB2C3rWwxmlznhqGWlHysdTgW1xt1wadQVBmhz86Z3etI5kdKSWbj99hyh9nCY9NYo7tP1RctQ6X5mUovHSb1vFMSg1DrSgFkN6zBEUnjCQpykjQ8B5axzFLQgg61SnG7hEt+aCSgUob1yIc4dSA/DGy6PPIViEQQrgKIXYKIS6l//zPbXUhxEtCiOMZXglCiE7p61YIIa5lWFc7O3kURfkvu479cH25IpGHA4n+Of89FWsqjtZ6em2djEyE1S/35oP1l+m3MoDAiJwdyM4cZPeMYCywW0pZAdid/vkRUsq9UsraUsrawMtAHPBXhk1GPVgvpTyezTyKomTCY9YqrNz13J21lJQb57WOY5buf/sZ0WfC8ejUgM+njubTdpU5eDkUn9n+LPG/QnJq/psS9IHsFoKOwMr09yuBTs/YviuwXUqZ/0usopgRna0DRWfOxJgEtwe8jUzO/vg7+UliwB7uLt6ETXEr3L74Hgu9jgHe5dg5ogVNyrkxbdt52n9zgGM3IrSOmiOyWwgKSSnvAqT/9HzG9t2B1Y8tmyqEOCmEmCOEsHrSjkKI/kKIACFEQEjIsx8EURTlUdZNXqXQe68Sdz2e0DG9tI5jNoxRYdwe9hE6AxRbtPKRsYSKu9jyfW8vFvWsS2RcMl0XHWLcplNExeXurGg57ZmFQAixSwhxOpNXx6w0JIQoAtQAdmRY/ClQGagPuAJjnrS/lHKJlNJLSunl4eGRlaYVRUnnPGIWTnUKEbrtBDHrFmgdxyzcG/wmiWGpFP10MBbla/1nvRCCttXTZkXr26QMa/65SavZ+9h8/HauzYqW055ZCKSUraWU1TN5bQaC0r/gH3zRBz/lUN2ATVLKh6VUSnlXpkkElgMNsvfHURTlaYROR+GF67Fy1XNn2gKSLxbs23KR88akDSjXtjr2b3301G3trQxMaF+VLUOaUdTZhmFrjtNr2T9cC837T25n99LQFuDBfG29gc1P2bYHj10WylBEBGn3F05nM4+iKM+gc3an2PxvkClwe2DvAjsERfye9dxbvBnbktZ4zPj5uferXsyJTYOaMrljNU7ciuSVuf7M23WJxJTnG8jOHGW3EEwHfIQQlwCf9M8IIbyEEA8HRBdClAZKAH6P7b9KCHEKOAW4A1OymUdRlOdg5fUyRT7qTvydJO4N6IA05t8eMZlJvnaGwFHjMdgKiv2wDmFlnaX99TrBu41Ls2ukNz5VCzFn10Xazd3PoSuhOZQ4Z6knixWlAAse9gZhO85SqEdzXCfmr3l4n8QYF83N15uREJRI6cVfY92sfbaPue9CMOM3n+ZWeDxd6hTjs9eq4Gb/xL4vmlFPFiuK8h8evmtxqOxE0Bp/YlbP0zpOjpNGI/f6tSf+ThJFR/Q0SREAaFnJk7+GezP4pXJsPXmHl339WPPPTYzGvPGLtioEilKACYOBoit+x8rdwO1pC0k4sFXrSDkqZORbRP0bhHv72ji+/7lJj21jqWfUK5XZNrQ5lQo5MHbjKbot/psL97I+K1puU4VAUQo4nbM7JZb9iM5ScPOjUSSdOqR1pBwRMf0jwrafxrlBUdxnrMqxdioUcmBN/0bMfKMml0NieG3+fqZvP098kvneTFaFQFEULCrWpeTC+WCEm++/n++GoYheMZ17K3diX8GBwkv/QOhy9qtPpxN0q1+CPSNb0qlOMRb5XcFnjh97zz+th712VCFQFAUAqwY+lJg5kZRYyc2eXUkNuqF1JJOIWT2P2zNXYF3IkmI/bctyD6HscLWzZNabtVjTvxFWBh19Vxxl0Kpj3IvKrKfIHwAAChRJREFUfFY0rahCoCjKQzav9KD4hMEkhaVw883XSb17XetI2RKz9hsCv1yIlbsFJddu1Ww+hkZl3dg2rDmftKnI7nPBtJ7tx/KD1x6ZFU1LqhAoivII+7c+ovjnA0gMTeZGt9dJCbyidaQXErP+WwK/+BZLNwMl125FX6iUpnmsDHqGvFyBvz5uQZ2Sznyx9Sydvj3IycBITXOBKgSKomTC/u2PKT5pCEnhKdx8qyMp185qHSlL7i+dTOCEb7B0TS8CRUprHemhUm52/PheA+b3qMPdqAQ6fXuQSVvOEJ2g3UB2qhAoipIp+25DKPHlcJKiUrj+5hskHtnx7J3MQPiUD7nt+wvWRaz/r737D46qugI4/j35BSFAEgkKBBOgDSMUx0qpECpQC6MYZ4A64EChaBUdcLCdgsUqzkBl2jq04mB1CqEgiFoESgEtFisFQSAWKz8KWDBAkAjyQ0iGEBKS7Okf71kjbMgj+yu7ez4zmdm372XvObubnL33vn2X3BVvkdSpa6RDuoKIMOyWTmyYOoixfXNZsr2EIXPeY91/TkTkQnZWCIwxDUq7dyK5L/4aX51SMuFnXFg1P9IhNUhrLnFq8ghOvrqJ1jdlkLN6Y7PqCfiTnprMrBG9WDWpP+3SWvDoax/x4OIdHDsb3iVb7BITxphG1fz33xx7cDzV5+q4YewdZD71UshPwbwWtcePcHzCKC4cvkBGfg4dCt/82roC0aC2zsfibSXM+cdBfKr8dHAeDw/oRnJi8J5nu8SEMabJkm/6Drlr3qH1N9ty8tVNfDbydupOfxbpsADnKqIlwwqoLKmgw4S76fjy+qgrAgBJiQlMGNCNd6cMYmBee2b//QD3vLCFHSVnQ962FQJjjCeJ7bPpvKaI6+/rz/mPz3LkniFUvr00YvFodRVnpo2lZPLTKJD7h2fIfHxOxOIJlk4ZqRSO78OC8X2oqKpl1LztPLFyD2WVoVte1IaGjDHXrHL96xx/ehY15yGzfw7tZ79MYlansLV/cdNf+HzGTKpO1tL25nbcMHdps5wUDtSF6lrmbviEhe8fIT01mekFPbi3dzbOEi7XzoaGjDFB0+quH9HtnffIHNiNc9uOcmjIDzj37GNodWi/MVt7ZD8nxg+hZOJ0aspryJ46huwV78dkEQBIa5HEUwU9eHPy7eS2a8XUFbvZdSz43zuwHoExJiAXNyzn5G9/w8XSalIyhOtGDiV94gwSWqcHrY2ag7v44vfTKdt6CPXBdYPyyJo1j8T22UFro7nz+ZSth84wIK/pa7Y31CMIqBCIyChgJtADuE1V/f53FpGhwFwgEfiTqn65kllXYBnOwvUfAT9W1UYHwqwQGNO8qM9HxSuzObPoNapO1ZKYqqTndyd93ERa9i9o0mP6Ksq58NcFlK1cQcXBcgDSe3cia9qvSLllQDDDjxuhKgQ9AB8wH3jcXyEQkUTgIM5SlqXADmCMqu4XkeXAKlVdJiLzgN2q+sfG2rVCYEzzpD4flasLObtkMRUHy0CF5LaQ1rMzrb7blxbfzifl1oEktGpzxe/VfX6USzs3c/FfW6jcvZcLxWVorZDYUskY0JOMidNI+Va/CGUWG0JSCOo9+CYaLgT5wExVvcvdftLd9SxwGuigqrWXH3c1VgiMaf5qSw9xfunzVBTtoPJwOb6aryY4E5KVxFRnW+ugrkrRuq/2J7eGtJtzaHNnAWkjJiCpaWGPPxY1VAiSwtB2NnCs3nYp0BdoB5Spam29+xsc8BORR4BHAHJyckITqTEmaJI6f4PMJ18kE9CqSqp3bqZ61zYuHf6EunPl1J2vABEkQUhMb0tyx44kd+tO6vfuJqlrz0iHH1caLQQi8i7Qwc+u6aq6xkMb/s5z0qvc75eqFgKF4PQIPLRrjGkmpGUrWuYPpWX+0EiHYvxotBCo6pAA2ygFbqy33Rk4DpwBMkQkye0VfHm/McaYMArH9wh2AHki0lVEUoDRwFp1Jic2AiPd4+4HvPQwjDHGBFFAhUBEfigipUA+8DcRWe/e30lE1gG4n/YnA+uBj4HlqrrPfYgngCkiUowzZ7AwkHiMMcZcO/tCmTHGxAm7xIQxxhi/rBAYY0ycs0JgjDFxzgqBMcbEuaicLBaR08DRJv56Fs53GOKJ5RwfLOfYF2i+uap6xeVLo7IQBEJEPvQ3ax7LLOf4YDnHvlDla0NDxhgT56wQGGNMnIvHQlAY6QAiwHKOD5Zz7AtJvnE3R2CMMebr4rFHYIwxph4rBMYYE+dithCIyFAROSAixSLySz/7W4jIG+7+D0SkS/ijDC4POU8Rkf0iskdENohIbiTiDKbGcq533EgRURGJ6lMNveQrIve5r/M+EXk93DEGm4f3dY6IbBSRne57uyAScQaTiCwSkVMisreB/SIiL7jPyR4R6R1Qg6oacz9AInAI6AakALuBnpcd8ygwz709Gngj0nGHIec7gFbu7UnxkLN7XBtgM1AE9Il03CF+jfOAnUCmu319pOMOQ86FwCT3dk+gJNJxByHvgUBvYG8D+wuAt3FWeuwHfBBIe7HaI7gNKFbVw6p6CVgGDL/smOHAEvf2SmCwiPhbPjNaNJqzqm5U1Up3swhnVbho5uV1BpgFzAaqwhlcCHjJ92HgJVU9B6Cqp8IcY7B5yVmBtu7tdGJgpUNV3Qycvcohw4FX1FGEs9pjx6a2F6uFIBs4Vm+71L3P7zHqLJ5TjrM4TrTyknN9D+F8oohmjeYsIrcCN6rqW+EMLES8vMbdge4islVEikQk2hcJ9pLzTGCcu0jWOuCx8IQWUdf6935Vja5ZHKX8fbK//DxZL8dEE8/5iMg4oA8wKKQRhd5VcxaRBOB54IFwBRRiXl7jJJzhoe/j9Pi2iEgvVS0LcWyh4iXnMcBiVX1ORPKBpW7OvtCHFzFB/f8Vqz2CUuDGetudubK7+P9jRCQJp0t5ta5Yc+clZ0RkCDAdGKaq1WGKLVQay7kN0AvYJCIlOGOpa6N4wtjr+3qNqtao6hHgAE5hiFZecn4IWA6gqtuBljgXZ4tlnv7evYrVQrADyBORriKSgjMZvPayY9YC97u3RwL/VHcWJko1mrM7TDIfpwhE+9gxNJKzqparapaqdlHVLjjzIsNUNVrXOfXyvl6Nc1IAIpKFM1R0OKxRBpeXnD8FBgOISA+cQnA6rFGG31pgvHv2UD+gXFVPNPXBYnJoSFVrRWQysB7nrINFqrpPRJ4BPlTVtcBCnC5kMU5PYHTkIg6cx5x/B7QGVrjz4p+q6rCIBR0gjznHDI/5rgfuFJH9QB3wC1X9InJRB8ZjzlOBBSLyc5zhkQei/EMdIvJnnOG9LHfuYwaQDKCq83DmQgqAYqAS+ElA7UX582WMMSZAsTo0ZIwxxiMrBMYYE+esEBhjTJyzQmCMMXHOCoExxsQ5KwTGGBPnrBAYY0yc+x9tMqL2TitCJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = sym.Symbol('x')\n",
    "f = sym.sin(2*sym.pi*x)\n",
    "psi, points = Lagrange_polynomials_01(x, N)\n",
    "Omega=[0, 1]\n",
    "u, c = least_squares(f, psi, Omega)\n",
    "comparison_plot(f, u, Omega)\n",
    "u, c = interpolation(f, psi, points)\n",
    "comparison_plot(f, u, Omega)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Figure](#fem:approx:global:Lagrange:fig:sine:ls:colloc) shows the results.\n",
    "There is a difference between the least squares and the interpolation\n",
    "technique but the difference decreases rapidly with  increasing $N$.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_ls_interp_sin_4.png, width=800]  Approximation via least squares (left) and interpolation (right) of a sine function by Lagrange interpolating polynomials of degree 3. <div id=\"fem:approx:global:Lagrange:fig:sine:ls:colloc\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:sine:ls:colloc\"></div>\n",
    "\n",
    "<p>Approximation via least squares (left) and interpolation (right) of a sine function by Lagrange interpolating polynomials of degree 3.</p>\n",
    "<img src=\"fig/Lagrange_ls_interp_sin_4.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "### Less successful example\n",
    "\n",
    "The next example concerns interpolating $f(x)=|1-2x|$ on $\\Omega\n",
    "=[0,1]$ using Lagrange polynomials. [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:7:14) shows a peculiar\n",
    "effect: the approximation starts to oscillate more and more as $N$\n",
    "grows. This numerical artifact is not surprising when looking at the\n",
    "individual Lagrange polynomials. [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:osc) shows two such\n",
    "polynomials, $\\psi_2(x)$ and $\\psi_7(x)$, both of degree 11 and\n",
    "computed from uniformly spaced points $x_{i}=i/11$,\n",
    "$i=0,\\ldots,11$, marked with circles.  We clearly see the property of\n",
    "Lagrange polynomials: $\\psi_2(x_{i})=0$ and $\\psi_7(x_{i})=0$ for\n",
    "all $i$, except $\\psi_2(x_{2})=1$ and $\\psi_7(x_{7})=1$.  The most\n",
    "striking feature, however, is the dominating oscillation near the\n",
    "boundary where $\\psi_2>5$ and $\\psi_7=-10$ in some points. The reason is easy to understand: since we force the\n",
    "functions to be zero at so many points, a polynomial of high degree is\n",
    "forced to oscillate between the points.  This is called\n",
    "*Runge's phenomenon* and you can read a more detailed explanation on\n",
    "[Wikipedia](http://en.wikipedia.org/wiki/Runge%27s_phenomenon).\n",
    "\n",
    "\n",
    "### Remedy for strong oscillations\n",
    "\n",
    "The oscillations can be reduced by a more clever choice of\n",
    "interpolation points, called the *Chebyshev nodes*:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto30\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "x_{i} = \\frac{1}{2} (a+b) + \\frac{1}{2}(b-a)\\cos\\left( \\frac{2i+1}{2(N+1)}pi\\right),\\quad i=0\\ldots,N,\n",
    "\\label{_auto30} \\tag{60}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "on the interval $\\Omega = [a,b]$.\n",
    "Here is a flexible version of the `Lagrange_polynomials_01` function above,\n",
    "valid for any interval $\\Omega =[a,b]$ and with the possibility to generate\n",
    "both uniformly distributed points and Chebyshev nodes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Lagrange_polynomials(x, N, Omega, point_distribution='uniform'):\n",
    "    if point_distribution == 'uniform':\n",
    "        if isinstance(x, sym.Symbol):\n",
    "            h = sym.Rational(Omega[1] - Omega[0], N)\n",
    "        else:\n",
    "            h = (Omega[1] - Omega[0])/float(N)\n",
    "        points = [Omega[0] + i*h for i in range(N+1)]\n",
    "    elif point_distribution == 'Chebyshev':\n",
    "        points = Chebyshev_nodes(Omega[0], Omega[1], N)\n",
    "    psi = [Lagrange_polynomial(x, i, points) for i in range(N+1)]\n",
    "    return psi, points\n",
    "\n",
    "def Chebyshev_nodes(a, b, N):\n",
    "    from math import cos, pi\n",
    "    return [0.5*(a+b) + 0.5*(b-a)*cos(float(2*i+1)/(2*(N+1))*pi) \\\n",
    "            for i in range(N+1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the functions computing Lagrange polynomials listed\n",
    "above are found in the module file `Lagrange.py`.\n",
    "\n",
    "[Figure](#fem:approx:global:Lagrange:fig:abs:Lag:Cheb:7:14) shows the\n",
    "improvement of using Chebyshev nodes, compared with the equidistant\n",
    "points in [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:7:14).  The reason for\n",
    "this improvement is that the corresponding Lagrange polynomials have\n",
    "much smaller oscillations, which can be seen by comparing [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:Cheb:osc) (Chebyshev\n",
    "points) with [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:osc) (equidistant\n",
    "points). Note the different scale on the vertical axes in the two\n",
    "figures and also that the Chebyshev points tend to cluster\n",
    "more around the element boundaries.\n",
    "\n",
    "\n",
    "Another cure for undesired oscillations of higher-degree interpolating\n",
    "polynomials is to use lower-degree Lagrange polynomials on many small\n",
    "patches of the domain. This is actually the idea pursued in the finite\n",
    "element method. For instance, linear Lagrange polynomials on $[0,1/2]$\n",
    "and $[1/2,1]$ would yield a perfect approximation to $f(x)=|1-2x|$ on\n",
    "$\\Omega = [0,1]$ since $f$ is piecewise linear.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_interp_abs_8_15.png, width=800]  Interpolation of an absolute value function by Lagrange polynomials and uniformly distributed interpolation points: degree 7 (left) and 14 (right).  <div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:7:14\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:7:14\"></div>\n",
    "\n",
    "<p>Interpolation of an absolute value function by Lagrange polynomials and uniformly distributed interpolation points: degree 7 (left) and 14 (right).</p>\n",
    "<img src=\"fig/Lagrange_interp_abs_8_15.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_basis_12.png, width=400]  Illustration of the oscillatory behavior of two Lagrange polynomials based on 12 uniformly spaced points (marked by circles).  <div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:osc\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:osc\"></div>\n",
    "\n",
    "<p>Illustration of the oscillatory behavior of two Lagrange polynomials based on 12 uniformly spaced points (marked by circles).</p>\n",
    "<img src=\"fig/Lagrange_basis_12.png\" width=400>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_interp_abs_Cheb_8_15.png, width=800]  Interpolation of an absolute value function by Lagrange polynomials and Chebyshev nodes as interpolation points: degree 7 (left) and 14 (right).  <div id=\"fem:approx:global:Lagrange:fig:abs:Lag:Cheb:7:14\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:abs:Lag:Cheb:7:14\"></div>\n",
    "\n",
    "<p>Interpolation of an absolute value function by Lagrange polynomials and Chebyshev nodes as interpolation points: degree 7 (left) and 14 (right).</p>\n",
    "<img src=\"fig/Lagrange_interp_abs_Cheb_8_15.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_basis_Cheb_12.png, width=400]  Illustration of the less oscillatory behavior of two Lagrange polynomials based on 12 Chebyshev points (marked by circles).  <div id=\"fem:approx:global:Lagrange:fig:abs:Lag:Cheb:osc\"></div> Note that the y-axis is different from [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:osc). -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:abs:Lag:Cheb:osc\"></div>\n",
    "\n",
    "<p>Illustration of the less oscillatory behavior of two Lagrange polynomials based on 12 Chebyshev points (marked by circles). Note that the y-axis is different from [Figure](#fem:approx:global:Lagrange:fig:abs:Lag:unif:osc).</p>\n",
    "<img src=\"fig/Lagrange_basis_Cheb_12.png\" width=400>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "How does the least squares or projection methods work with Lagrange\n",
    "polynomials?\n",
    "We can just call the `least_squares` function, but\n",
    "`sympy` has problems integrating the $f(x)=|1-2x|$\n",
    "function times a polynomial, so we need to fall back on numerical\n",
    "integration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares(f, psi, Omega):\n",
    "    N = len(psi) - 1\n",
    "    A = sym.zeros(N+1, N+1)\n",
    "    b = sym.zeros(N+1, 1)\n",
    "    x = sym.Symbol('x')\n",
    "    for i in range(N+1):\n",
    "        for j in range(i, N+1):\n",
    "            integrand = psi[i]*psi[j]\n",
    "            I = sym.integrate(integrand, (x, Omega[0], Omega[1]))\n",
    "            if isinstance(I, sym.Integral):\n",
    "                # Could not integrate symbolically, fall back\n",
    "                # on numerical integration with mpmath.quad\n",
    "                integrand = sym.lambdify([x], integrand, 'mpmath')\n",
    "                I = mpmath.quad(integrand, [Omega[0], Omega[1]])\n",
    "            A[i,j] = A[j,i] = I\n",
    "        integrand = psi[i]*f\n",
    "        I = sym.integrate(integrand, (x, Omega[0], Omega[1]))\n",
    "        if isinstance(I, sym.Integral):\n",
    "            integrand = sym.lambdify([x], integrand, 'mpmath')\n",
    "            I = mpmath.quad(integrand, [Omega[0], Omega[1]])\n",
    "        b[i,0] = I\n",
    "    c = A.LUsolve(b)\n",
    "    c = [sym.simplify(c[i,0]) for i in range(c.shape[0])]\n",
    "    u = sum(c[i]*psi[i] for i in range(len(psi)))\n",
    "    return u, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Convergence of Lagrange polynomials. -->\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_ls_abs_12.png, width=400]  Illustration of an approximation of the absolute value function using the least square method .  <div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:ls\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:global:Lagrange:fig:abs:Lag:unif:ls\"></div>\n",
    "\n",
    "<p>Illustration of an approximation of the absolute value function using the least square method .</p>\n",
    "<img src=\"fig/Lagrange_ls_abs_12.png\" width=400>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Bernstein polynomials\n",
    "<div id=\"fem:approx:global:Bernstein\"></div>\n",
    "\n",
    "An alternative to the Taylor and Lagrange families of polynomials\n",
    "are the Bernstein polynomials.\n",
    "These polynomials are popular in visualization and we include a\n",
    "presentation of them for completeness. Furthermore, as we\n",
    "will demonstrate, the choice of basis functions may matter\n",
    "in terms of accuracy and efficiency.\n",
    "In fact, in finite element methods,\n",
    "a main challenge, from a numerical analysis point of view,\n",
    "is to determine appropriate basis functions for\n",
    "a particular purpose or equation.\n",
    "\n",
    "On the unit interval, the Bernstein\n",
    "polynomials are defined in terms of powers of $x$ and $1-x$\n",
    "(the barycentric coordinates of the unit interval) as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"bernstein:basis\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "B_{i,n} = \\binom{n}{i} x^i (1-x)^{n-i}, \\quad i=0, \\ldots, n .\n",
    "\\label{bernstein:basis} \\tag{61}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- dom:FIGURE: [fig/Bernstein_basis8.png] The nine functions of a Bernstein basis of order 8. <div id=\"Bernstein_basis_8\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"Bernstein_basis_8\"></div>\n",
    "\n",
    "<p>The nine functions of a Bernstein basis of order 8.</p>\n",
    "<img src=\"fig/Bernstein_basis8.png\" >\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "<!-- dom:FIGURE: [fig/Lagrange_basis8.png] The nine functions of a Lagrange basis of order 8. <div id=\"Lagrange_basis_8\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"Lagrange_basis_8\"></div>\n",
    "\n",
    "<p>The nine functions of a Lagrange basis of order 8.</p>\n",
    "<img src=\"fig/Lagrange_basis8.png\" >\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "The [Figure](#Bernstein_basis_8) shows the  basis functions of a Bernstein basis of order 8.\n",
    "This figure should be compared against [Figure](#Lagrange_basis_8), which\n",
    "shows the corresponding Lagrange basis of order 8.\n",
    "The Lagrange basis is convenient because it is a nodal basis, that is; the basis functions are 1 in their nodal points and zero at all other nodal points as described by ([57](#fem:inter:prop)).\n",
    "However, looking at [Figure](#Lagrange_basis_8)\n",
    "we also notice that the basis function are oscillatory and have absolute\n",
    "values that are significantly larger than 1 between the nodal points.\n",
    "Consider for instance the basis function represented by the purple color.\n",
    "It is 1 at $x=0.5$ and 0 at all other nodal points\n",
    "and hence this basis function represents the value at the mid-point.\n",
    "However, this function also has strong\n",
    "negative contributions close to the element boundaries where\n",
    "it takes negative values lower than $-2$.\n",
    "For the Bernstein basis, all  functions are positive and\n",
    "all functions output values in $[0,1]$. Therefore there is no oscillatory behavior.\n",
    "The main disadvantage of the Bernstein basis is that the basis is not\n",
    "a nodal basis. In fact, all functions contribute everywhere except $x=0$ and $x=1$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice.**\n",
    "\n",
    "We have considered approximation with a sinusoidal basis and with  Lagrange or Bernstein polynomials,\n",
    "all of which are frequently used for scientific computing. The Lagrange polynomials (of various\n",
    "order) are extensively used in finite element methods, while the Bernstein polynomials are more used\n",
    "in computational geometry. The Lagrange and the Bernstein families are, however, but a few in the jungle of polynomial\n",
    "spaces used for finite element computations and their efficiency and accuracy can vary quite substantially.\n",
    "Furthermore, while a method may be efficient and accurate for one type of PDE it might not even converge for\n",
    "another type of PDE.  The development and analysis of finite element methods for different purposes is currently an intense research field\n",
    "and has been so for several decades.\n",
    "FEniCS has implemented a wide range of polynomial spaces and has a general\n",
    "framework for implementing new elements. While finite element methods\n",
    "explore different families of polynomials, the so-called spectral methods explore the use\n",
    "of sinusoidal functions or polynomials with high order. From an abstract point of view, the different methods\n",
    "can all be obtained simply by changing the basis for the trial and test functions. However, their\n",
    "efficiency and accuracy may vary substantially."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Approximation of functions in higher dimensions\n",
    "<div id=\"fem:approx:2D\"></div>\n",
    "\n",
    "All the concepts and algorithms developed for approximation of 1D functions\n",
    "$f(x)$ can readily be extended to 2D functions $f(x,y)$ and 3D functions\n",
    "$f(x,y,z)$. Basically, the extensions consist of defining basis functions\n",
    "${\\psi}_i(x,y)$ or ${\\psi}_i(x,y,z)$ over some domain $\\Omega$, and\n",
    "for the least squares and Galerkin methods, the integration is done over\n",
    "$\\Omega$.\n",
    "\n",
    "As in 1D, the least squares and projection/Galerkin methods\n",
    "lead to linear systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "\\sum_{j\\in{\\mathcal{I}_s}} A_{i,j}c_j &= b_i,\\quad i\\in{\\mathcal{I}_s},\\\\\n",
    "A_{i,j} &= ({\\psi}_i,{\\psi}_j),\\\\\n",
    "b_i &= (f,{\\psi}_i),\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where the inner product of two functions $f(x,y)$ and $g(x,y)$ is defined\n",
    "completely analogously to the 1D case ([25](#fem:approx:LS:innerprod)):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto31\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "(f,g) = \\int_\\Omega f(x,y)g(x,y) dx dy .\n",
    "\\label{_auto31} \\tag{64}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2D basis functions as tensor products of 1D functions\n",
    "<div id=\"fem:approx:2D:global\"></div>\n",
    "\n",
    "\n",
    "\n",
    "One straightforward way to construct a basis in 2D is to combine 1D\n",
    "basis functions. Say we have the 1D vector space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:2D:Vx\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "V_x = \\mbox{span}\\{ \\hat{\\psi}_0(x),\\ldots,\\hat{\\psi}_{N_x}(x)\\}\n",
    "\\label{fem:approx:2D:Vx} \\tag{65}\n",
    "{\\thinspace .}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar space for a function's variation in $y$ can be defined,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"fem:approx:2D:Vy\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "V_y = \\mbox{span}\\{ \\hat{\\psi}_0(y),\\ldots,\\hat{\\psi}_{N_y}(y)\\}\n",
    "\\label{fem:approx:2D:Vy} \\tag{66}\n",
    "{\\thinspace .}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then form 2D basis functions as *tensor products* of 1D basis functions.\n",
    "\n",
    "**Tensor products.**\n",
    "\n",
    "Given two vectors $a=(a_0,\\ldots,a_M)$ and $b=(b_0,\\ldots,b_N)$,\n",
    "their *outer tensor product*, also called the *dyadic product*,\n",
    "is $p=a\\otimes b$, defined through"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "p_{i,j}=a_ib_j,\\quad i=0,\\ldots,M,\\ j=0,\\ldots,N{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the tensor terminology,\n",
    "$a$ and $b$ are first-order tensors (vectors with one index, also termed\n",
    "rank-1 tensors), and then their outer\n",
    "tensor product is a second-order tensor (matrix with two indices, also\n",
    "termed rank-2 tensor). The\n",
    "corresponding *inner tensor product* is the well-known scalar or dot\n",
    "product of two vectors: $p=a\\cdot b = \\sum_{j=0}^N a_jb_j$. Now,\n",
    "$p$ is a rank-0 tensor.\n",
    "\n",
    "\n",
    "Tensors are typically represented by arrays in computer code.\n",
    "In the above example, $a$ and $b$ are represented by\n",
    "one-dimensional arrays of length\n",
    "$M$ and $N$, respectively, while $p=a\\otimes b$ must be represented\n",
    "by a two-dimensional array of size $M\\times N$.\n",
    "\n",
    "[Tensor products](http://en.wikipedia.org/wiki/Tensor_product) can\n",
    "be used in a variety of contexts.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<!-- The following is from <http://en.wikipedia.org/wiki/Tensor_product>, -->\n",
    "<!-- Notation and examples -->\n",
    "Given the vector spaces $V_x$ and $V_y$ as defined\n",
    "in ([65](#fem:approx:2D:Vx)) and ([66](#fem:approx:2D:Vy)), the\n",
    "tensor product space $V=V_x\\otimes V_y$ has a basis formed\n",
    "as the tensor product of the basis for $V_x$ and $V_y$.\n",
    "That is, if $\\left\\{ {\\psi}_i(x) \\right\\}_{i\\in{\\mathcal{I}_x}}$\n",
    "and $\\left\\{ {\\psi}_i(y) \\right\\}_{i\\in {\\mathcal{I}_y}}$ are basis for\n",
    "$V_x$ and $V_y$, respectively, the elements in the basis for $V$ arise\n",
    "from the tensor product:\n",
    "$\\left\\{ {\\psi}_i(x){\\psi}_j(y) \\right\\}_{i\\in {\\mathcal{I}_x},j\\in {\\mathcal{I}_y}}$.\n",
    "The index sets are $I_x=\\{0,\\ldots,N_x\\}$ and $I_y=\\{0,\\ldots,N_y\\}$.\n",
    "\n",
    "The notation for a basis function in 2D can employ a double index as in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_{p,q}(x,y) = \\hat{\\psi}_p(x)\\hat{\\psi}_q(y),\n",
    "\\quad p\\in{\\mathcal{I}_x},q\\in{\\mathcal{I}_y}{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expansion for $u$ is then written as a double sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "u = \\sum_{p\\in{\\mathcal{I}_x}}\\sum_{q\\in{\\mathcal{I}_y}} c_{p,q}{\\psi}_{p,q}(x,y){\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we may employ a single index,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_i(x,y) = \\hat{\\psi}_p(x)\\hat{\\psi}_q(y),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and use the standard form for $u$,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "u = \\sum_{j\\in{\\mathcal{I}_s}} c_j{\\psi}_j(x,y){\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The single index can be expressed in terms of the double index through\n",
    "$i=p (N_y+1) + q$ or $i=q (N_x+1) + p$.\n",
    "\n",
    "## Example on polynomial basis in 2D\n",
    "\n",
    "Let us again consider an approximation with the least squares method, but now\n",
    "with basis functions in 2D.\n",
    "Suppose we choose $\\hat{\\psi}_p(x)=x^p$, and try an approximation with\n",
    "$N_x=N_y=1$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_{0,0}=1,\\quad {\\psi}_{1,0}=x, \\quad {\\psi}_{0,1}=y,\n",
    "\\quad {\\psi}_{1,1}=xy\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a mapping to one index like $i=q (N_x+1) + p$, we get"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_0=1,\\quad {\\psi}_1=x, \\quad {\\psi}_2=y,\\quad{\\psi}_3 =xy\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the specific choice $f(x,y) = (1+x^2)(1+2y^2)$ on\n",
    "$\\Omega = [0,L_x]\\times [0,L_y]$, we can perform actual calculations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "A_{0,0} &= ({\\psi}_0,{\\psi}_0) = \\int_0^{L_y}\\int_{0}^{L_x} 1 dxdy = L_{x} L_{y}, \\\\\n",
    "A_{0,1} &= ({\\psi}_0,{\\psi}_1) = \\int_0^{L_y}\\int_{0}^{L_x} x dxdy = \\frac{L_{x}^{2} L_{y}}{2}, \\\\\n",
    "A_{0,2} &= ({\\psi}_0,{\\psi}_2) = \\int_0^{L_y}\\int_{0}^{L_x} y dxdy = \\frac{L_{x} L_{y}^{2}}{2}, \\\\\n",
    "A_{0,3} &= ({\\psi}_0,{\\psi}_3) = \\int_0^{L_y}\\int_{0}^{L_x} x y dxdy = \\frac{L_{x}^{2} L_{y}^{2}}{4}, \\\\\n",
    "A_{1,0} &= ({\\psi}_1,{\\psi}_0) = \\int_0^{L_y}\\int_{0}^{L_x} x dxdy = \\frac{L_{x}^{2} L_{y}}{2}, \\\\\n",
    "A_{1,1} &= ({\\psi}_1,{\\psi}_1) = \\int_0^{L_y}\\int_{0}^{L_x} x^{2} dxdy = \\frac{L_{x}^{3} L_{y}}{3}, \\\\\n",
    "A_{1,2} &= ({\\psi}_1,{\\psi}_2) = \\int_0^{L_y}\\int_{0}^{L_x} x y dxdy = \\frac{L_{x}^{2} L_{y}^{2}}{4}, \\\\\n",
    "A_{1,3} &= ({\\psi}_1,{\\psi}_3) = \\int_0^{L_y}\\int_{0}^{L_x} x^{2} y dxdy = \\frac{L_{x}^{3} L_{y}^{2}}{6}, \\\\\n",
    "A_{2,0} &= ({\\psi}_2,{\\psi}_0) = \\int_0^{L_y}\\int_{0}^{L_x} y dxdy = \\frac{L_{x} L_{y}^{2}}{2}, \\\\\n",
    "A_{2,1} &= ({\\psi}_2,{\\psi}_1) = \\int_0^{L_y}\\int_{0}^{L_x} x y dxdy = \\frac{L_{x}^{2} L_{y}^{2}}{4}, \\\\\n",
    "A_{2,2} &= ({\\psi}_2,{\\psi}_2) = \\int_0^{L_y}\\int_{0}^{L_x} y^{2} dxdy = \\frac{L_{x} L_{y}^{3}}{3}, \\\\\n",
    "A_{2,3} &= ({\\psi}_2,{\\psi}_3) = \\int_0^{L_y}\\int_{0}^{L_x} x y^{2} dxdy = \\frac{L_{x}^{2} L_{y}^{3}}{6}, \\\\\n",
    "A_{3,0} &= ({\\psi}_3,{\\psi}_0) = \\int_0^{L_y}\\int_{0}^{L_x} x y dxdy = \\frac{L_{x}^{2} L_{y}^{2}}{4}, \\\\\n",
    "A_{3,1} &= ({\\psi}_3,{\\psi}_1) = \\int_0^{L_y}\\int_{0}^{L_x} x^{2} y dxdy = \\frac{L_{x}^{3} L_{y}^{2}}{6}, \\\\\n",
    "A_{3,2} &= ({\\psi}_3,{\\psi}_2) = \\int_0^{L_y}\\int_{0}^{L_x} x y^{2} dxdy = \\frac{L_{x}^{2} L_{y}^{3}}{6}, \\\\\n",
    "A_{3,3} &= ({\\psi}_3,{\\psi}_3) = \\int_0^{L_y}\\int_{0}^{L_x} x^{2} y^{2} dxdy = \\frac{L_{x}^{3} L_{y}^{3}}{9}\n",
    "{\\thinspace .}\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The right-hand side vector has the entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "b_{0} &= ({\\psi}_0,f) = \\int_0^{L_y}\\int_{0}^{L_x}1\\cdot (1+x^2)(1+2y^2) dxdy\\\\\n",
    "&= \\int_0^{L_y}(1+2y^2)dy \\int_{0}^{L_x} (1+x^2)dx\n",
    "= (L_y + \\frac{2}{3}L_y^3)(L_x + \\frac{1}{3}L_x^3)\\\\\n",
    "b_{1} &= ({\\psi}_1,f) = \\int_0^{L_y}\\int_{0}^{L_x} x(1+x^2)(1+2y^2) dxdy\\\\\n",
    "&=\\int_0^{L_y}(1+2y^2)dy \\int_{0}^{L_x} x(1+x^2)dx\n",
    "= (L_y + \\frac{2}{3}L_y^3)({\\frac{1}{2}}L_x^2 + \\frac{1}{4}L_x^4)\\\\\n",
    "b_{2} &= ({\\psi}_2,f) = \\int_0^{L_y}\\int_{0}^{L_x} y(1+x^2)(1+2y^2) dxdy\\\\\n",
    "&= \\int_0^{L_y}y(1+2y^2)dy \\int_{0}^{L_x} (1+x^2)dx\n",
    "= ({\\frac{1}{2}}L_y^2 + {\\frac{1}{2}}L_y^4)(L_x + \\frac{1}{3}L_x^3)\\\\\n",
    "b_{3} &= ({\\psi}_3,f) = \\int_0^{L_y}\\int_{0}^{L_x} xy(1+x^2)(1+2y^2) dxdy\\\\\n",
    "&= \\int_0^{L_y}y(1+2y^2)dy \\int_{0}^{L_x} x(1+x^2)dx\n",
    "= ({\\frac{1}{2}}L_y^2 + {\\frac{1}{2}}L_y^4)({\\frac{1}{2}}L_x^2 + \\frac{1}{4}L_x^4)\n",
    "{\\thinspace .}\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a general pattern in these calculations that we can explore.\n",
    "An arbitrary matrix entry has the formula"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "A_{i,j} &= ({\\psi}_i,{\\psi}_j) = \\int_0^{L_y}\\int_{0}^{L_x}\n",
    "{\\psi}_i{\\psi}_j dx dy \\\\\n",
    "&= \\int_0^{L_y}\\int_{0}^{L_x}\n",
    "{\\psi}_{p,q}{\\psi}_{r,s} dx dy\n",
    "= \\int_0^{L_y}\\int_{0}^{L_x}\n",
    "\\hat{\\psi}_p(x)\\hat{\\psi}_q(y)\\hat{\\psi}_r(x)\\hat{\\psi}_s(y) dx dy\\\\\n",
    "&= \\int_0^{L_y} \\hat{\\psi}_q(y)\\hat{\\psi}_s(y)dy\n",
    "\\int_{0}^{L_x} \\hat{\\psi}_p(x) \\hat{\\psi}_r(x) dx\\\\\n",
    "&= \\hat A^{(x)}_{p,r}\\hat A^{(y)}_{q,s},\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat A^{(x)}_{p,r} = \\int_{0}^{L_x} \\hat{\\psi}_p(x) \\hat{\\psi}_r(x) dx,\n",
    "\\quad\n",
    "\\hat A^{(y)}_{q,s} = \\int_0^{L_y} \\hat{\\psi}_q(y)\\hat{\\psi}_s(y)dy,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "are matrix entries for one-dimensional approximations. Moreover,\n",
    "$i=p N_x+q$ and $j=s N_y+r$.\n",
    "\n",
    "\n",
    "With $\\hat{\\psi}_p(x)=x^p$ we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat A^{(x)}_{p,r} = \\frac{1}{p+r+1}L_x^{p+r+1},\\quad\n",
    "\\hat A^{(y)}_{q,s} = \\frac{1}{q+s+1}L_y^{q+s+1},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "A_{i,j} = \\hat A^{(x)}_{p,r} \\hat A^{(y)}_{q,s} =\n",
    "\\frac{1}{p+r+1}L_x^{p+r+1} \\frac{1}{q+s+1}L_y^{q+s+1},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for $p,r\\in{\\mathcal{I}_x}$ and $q,s\\in{\\mathcal{I}_y}$.\n",
    "\n",
    "Corresponding reasoning for the right-hand side leads to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "b_i &= ({\\psi}_i,f) = \\int_0^{L_y}\\int_{0}^{L_x}{\\psi}_i f\\,dxdx\\\\\n",
    "&= \\int_0^{L_y}\\int_{0}^{L_x}\\hat{\\psi}_p(x)\\hat{\\psi}_q(y) f\\,dxdx\\\\\n",
    "&= \\int_0^{L_y}\\hat{\\psi}_q(y) (1+2y^2)dy\n",
    "\\int_0^{L_y}\\hat{\\psi}_p(x) (1+x^2)dx\\\\\n",
    "&= \\int_0^{L_y} y^q (1+2y^2)dy\n",
    "\\int_0^{L_y}x^p (1+x^2)dx\\\\\n",
    "&= (\\frac{1}{q+1} L_y^{q+1} + \\frac{2}{q+3}L_y^{q+3})\n",
    "(\\frac{1}{p+1} L_x^{p+1} + \\frac{1}{p+3}L_x^{p+3})\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choosing $L_x=L_y=2$, we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "A =\n",
    "\\left[\\begin{array}{cccc}\n",
    "4 & 4 & 4 & 4\\\\\n",
    "4 & \\frac{16}{3} & 4 & \\frac{16}{3}\\\\\n",
    "4 & 4 & \\frac{16}{3} & \\frac{16}{3}\\\\\n",
    "4 & \\frac{16}{3} & \\frac{16}{3} & \\frac{64}{9}\n",
    "\\end{array}\\right],\\quad\n",
    "b = \\left[\\begin{array}{c}\n",
    "\\frac{308}{9}\\\\\\frac{140}{3}\\\\44\\\\60\\end{array}\\right],\n",
    "\\quad c = \\left[\n",
    "\\begin{array}{r}\n",
    "-\\frac{1}{9}\\\\\n",
    "- \\frac{2}{3}\n",
    "\\frac{4}{3} \\\\\n",
    " 8\n",
    "\\end{array}\\right]\n",
    "{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Figure](#fem:approx:fe:2D:fig:ubilinear) illustrates the result.\n",
    "\n",
    "<!-- dom:FIGURE: [fig/approx2D_bilinear.png, width=800] Approximation of a 2D quadratic function (left) by a 2D bilinear function (right) using the Galerkin or least squares method. <div id=\"fem:approx:fe:2D:fig:ubilinear\"></div> -->\n",
    "<!-- begin figure -->\n",
    "<div id=\"fem:approx:fe:2D:fig:ubilinear\"></div>\n",
    "\n",
    "<p>Approximation of a 2D quadratic function (left) by a 2D bilinear function (right) using the Galerkin or least squares method.</p>\n",
    "<img src=\"fig/approx2D_bilinear.png\" width=800>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "The calculations can also be done using  `sympy`. The following code computes the matrix of our example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sympy as sym\n",
    "\n",
    "x, y, Lx, Ly = sym.symbols(\"x y L_x L_y\")\n",
    "\n",
    "def integral(integrand):\n",
    "  Ix = sym.integrate(integrand, (x, 0, Lx))\n",
    "  I = sym.integrate(Ix, (y, 0, Ly))\n",
    "  return I\n",
    "\n",
    "basis = [1, x, y, x*y]\n",
    "A = sym.Matrix(sym.zeros(4,4))\n",
    "\n",
    "for i in range(len(basis)):\n",
    "  psi_i = basis[i]\n",
    "  for j in range(len(basis)):\n",
    "    psi_j = basis[j]\n",
    "    A[i,j] = integral(psi_i*psi_j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remark that `sympy` may even write the output in LaTeX or C++ format\n",
    "using the functions  `latex` and  `ccode`.\n",
    "\n",
    "## Implementation\n",
    "<div id=\"fem:approx:2D:global:code\"></div>\n",
    "\n",
    "The `least_squares` function from\n",
    "the section [Orthogonal basis functions](#fem:approx:global:orth) and/or the\n",
    "file [`approx1D.py`](src/fe_approx1D.py)\n",
    "can with very small modifications solve 2D approximation problems.\n",
    "First, let `Omega` now be a list of the intervals in $x$ and $y$ direction.\n",
    "For example, $\\Omega = [0,L_x]\\times [0,L_y]$ can be represented\n",
    "by `Omega = [[0, L_x], [0, L_y]]`.\n",
    "\n",
    "Second, the symbolic integration must be extended to 2D:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT RUN THIS CELL, THIS IS A DEMONSTRATION\n",
    "integrand = psi[i]*psi[j]\n",
    "I = sym.integrate(integrand,\n",
    "                 (x, Omega[0][0], Omega[0][1]),\n",
    "                 (y, Omega[1][0], Omega[1][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "provided `integrand` is an expression involving the `sympy` symbols `x`\n",
    "and `y`.\n",
    "The 2D version of numerical integration becomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT RUN THIS CELL, THIS IS A DEMONSTRATION\n",
    "if isinstance(I, sym.Integral):\n",
    "    integrand = sym.lambdify([x,y], integrand, 'mpmath')\n",
    "    I = mpmath.quad(integrand,\n",
    "                    [Omega[0][0], Omega[0][1]],\n",
    "                    [Omega[1][0], Omega[1][1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The right-hand side integrals are modified in a similar way.\n",
    "(We should add that `mpmath.quad` is sufficiently fast\n",
    "even in 2D, but `scipy.integrate.nquad` is much faster.)\n",
    "\n",
    "Third, we must construct a list of 2D basis functions. Here are two\n",
    "examples based on tensor products of 1D \"Taylor-style\" polynomials $x^i$\n",
    "and 1D sine functions $\\sin((i+1)\\pi x)$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def taylor(x, y, Nx, Ny):\n",
    "    return [x**i*y**j for i in range(Nx+1) for j in range(Ny+1)]\n",
    "\n",
    "def sines(x, y, Nx, Ny):\n",
    "    return [sym.sin(sym.pi*(i+1)*x)*sym.sin(sym.pi*(j+1)*y)\n",
    "            for i in range(Nx+1) for j in range(Ny+1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complete code appears in\n",
    "[`approx2D.py`](src/fe_approx2D.py).\n",
    "\n",
    "The previous hand calculation where a quadratic $f$ was approximated by\n",
    "a bilinear function can be computed symbolically by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...evaluating matrix...\n",
      "(0,0)\n",
      "(0,1)\n",
      "(0,2)\n",
      "(0,3)\n",
      "(1,1)\n",
      "(1,2)\n",
      "(1,3)\n",
      "(2,2)\n",
      "(2,3)\n",
      "(3,3)\n",
      "\n",
      "('A:\\n', Matrix([\n",
      "[4,    4,    4,    4],\n",
      "[4, 16/3,    4, 16/3],\n",
      "[4,    4, 16/3, 16/3],\n",
      "[4, 16/3, 16/3, 64/9]]), '\\nb:\\n', Matrix([\n",
      "[308/9],\n",
      "[140/3],\n",
      "[   44],\n",
      "[   60]]))\n",
      "('coeff:', [-1/9, 4/3, -2/3, 8])\n",
      "('approximation:', 8*x*y - 2*x/3 + 4*y/3 - 1/9)\n",
      "('f:', 2*x**2*y**2 + x**2 + 2*y**2 + 1)\n",
      "8*x*y - 2*x/3 + 4*y/3 - 1/9\n"
     ]
    }
   ],
   "source": [
    "from src.approx2D import *\n",
    "\n",
    "f = (1+x**2)*(1+2*y**2)\n",
    "psi = taylor(x, y, 1, 1)\n",
    "Omega = [[0, 2], [0, 2]]\n",
    "u = least_squares(f, psi, Omega)\n",
    "print(u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2*x**2*y**2 + x**2 + 2*y**2 + 1\n"
     ]
    }
   ],
   "source": [
    "print((sym.expand(f)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may continue with adding higher powers to the basis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...evaluating matrix...\n",
      "(0,0)\n",
      "(0,1)\n",
      "(0,2)\n",
      "(0,3)\n",
      "(0,4)\n",
      "(0,5)\n",
      "(0,6)\n",
      "(0,7)\n",
      "(0,8)\n",
      "(1,1)\n",
      "(1,2)\n",
      "(1,3)\n",
      "(1,4)\n",
      "(1,5)\n",
      "(1,6)\n",
      "(1,7)\n",
      "(1,8)\n",
      "(2,2)\n",
      "(2,3)\n",
      "(2,4)\n",
      "(2,5)\n",
      "(2,6)\n",
      "(2,7)\n",
      "(2,8)\n",
      "(3,3)\n",
      "(3,4)\n",
      "(3,5)\n",
      "(3,6)\n",
      "(3,7)\n",
      "(3,8)\n",
      "(4,4)\n",
      "(4,5)\n",
      "(4,6)\n",
      "(4,7)\n",
      "(4,8)\n",
      "(5,5)\n",
      "(5,6)\n",
      "(5,7)\n",
      "(5,8)\n",
      "(6,6)\n",
      "(6,7)\n",
      "(6,8)\n",
      "(7,7)\n",
      "(7,8)\n",
      "(8,8)\n",
      "\n",
      "('A:\\n', Matrix([\n",
      "[   4,    4,   16/3,    4,    4,   16/3,   16/3,   16/3,    64/9],\n",
      "[   4, 16/3,      8,    4, 16/3,      8,   16/3,   64/9,    32/3],\n",
      "[16/3,    8,   64/5, 16/3,    8,   64/5,   64/9,   32/3,  256/15],\n",
      "[   4,    4,   16/3, 16/3, 16/3,   64/9,      8,      8,    32/3],\n",
      "[   4, 16/3,      8, 16/3, 64/9,   32/3,      8,   32/3,      16],\n",
      "[16/3,    8,   64/5, 64/9, 32/3, 256/15,   32/3,     16,   128/5],\n",
      "[16/3, 16/3,   64/9,    8,    8,   32/3,   64/5,   64/5,  256/15],\n",
      "[16/3, 64/9,   32/3,    8, 32/3,     16,   64/5, 256/15,   128/5],\n",
      "[64/9, 32/3, 256/15, 32/3,   16,  128/5, 256/15,  128/5, 1024/25]]), '\\nb:\\n', Matrix([\n",
      "[    308/9],\n",
      "[    140/3],\n",
      "[  3248/45],\n",
      "[       44],\n",
      "[       60],\n",
      "[    464/5],\n",
      "[  2992/45],\n",
      "[    272/3],\n",
      "[31552/225]]))\n",
      "('coeff:', [1, 0, 2, 0, 0, 0, 1, 0, 2])\n",
      "('approximation:', 2*x**2*y**2 + x**2 + 2*y**2 + 1)\n",
      "('f:', 2*x**2*y**2 + x**2 + 2*y**2 + 1)\n",
      "2*x**2*y**2 + x**2 + 2*y**2 + 1\n"
     ]
    }
   ],
   "source": [
    "psi = taylor(x, y, 2, 2)\n",
    "u = least_squares(f, psi, Omega)\n",
    "print(u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2*x**2*y**2 + x**2 + 2*y**2 - (x**2 + 1)*(2*y**2 + 1) + 1\n"
     ]
    }
   ],
   "source": [
    "print((u-f))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For $N_x\\geq 2$ and $N_y\\geq 2$ we recover the exact function $f$, as\n",
    "expected, since in that case $f\\in V$, see\n",
    "the section [Perfect approximation](#fem:approx:global:exact1).\n",
    "\n",
    "## Extension to 3D\n",
    "<div id=\"fem:approx:3D:global\"></div>\n",
    "\n",
    "Extension to 3D is in principle straightforward once the 2D extension\n",
    "is understood. The only major difference is that we need the\n",
    "repeated outer tensor product,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "V = V_x\\otimes V_y\\otimes V_z{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, given vectors (first-order tensors)\n",
    "$a^{(q)} = (a^{(q)}_0,\\ldots,a^{(q)}_{N_q})$, $q=0,\\ldots,m$,\n",
    "the tensor product $p=a^{(0)}\\otimes\\cdots\\otimes a^{m}$ has\n",
    "elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "p_{i_0,i_1,\\ldots,i_m} = a^{(0)}_{i_1}a^{(1)}_{i_1}\\cdots a^{(m)}_{i_m}{\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basis functions in 3D are then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "{\\psi}_{p,q,r}(x,y,z) = \\hat{\\psi}_p(x)\\hat{\\psi}_q(y)\\hat{\\psi}_r(z),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with $p\\in{\\mathcal{I}_x}$, $q\\in{\\mathcal{I}_y}$, $r\\in{\\mathcal{I}_z}$. The expansion of $u$ becomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "u(x,y,z) = \\sum_{p\\in{\\mathcal{I}_x}}\\sum_{q\\in{\\mathcal{I}_y}}\\sum_{r\\in{\\mathcal{I}_z}} c_{p,q,r}\n",
    "{\\psi}_{p,q,r}(x,y,z){\\thinspace .}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single index can be introduced also here, e.g., $i=rN_xN_y + qN_x + p$,\n",
    "$u=\\sum_i c_i{\\psi}_i(x,y,z)$.\n",
    "\n",
    "**Use of tensor product spaces.**\n",
    "\n",
    "Constructing a multi-dimensional space and basis from tensor products\n",
    "of 1D spaces is a standard technique when working with global basis\n",
    "functions. In the world of finite elements, constructing basis functions\n",
    "by tensor products is much used on quadrilateral and hexahedra cell\n",
    "shapes, but not on triangles and tetrahedra. Also, the global\n",
    "finite element basis functions are almost exclusively denoted by a single\n",
    "index and not by the natural tuple of indices that arises from\n",
    "tensor products."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}