notebooks/1_MoreFundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Python Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Notebook settings\n",
    "\n",
    "# multiple lines of output per cell\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### Today\n",
    ">\n",
    ">- Writing Functions\n",
    ">\n",
    ">\n",
    ">- Handling Exceptions\n",
    ">\n",
    ">\n",
    ">- Classes\n",
    ">\n",
    ">\n",
    ">- File Input/Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A little bit of theory: Assignment & Data types\n",
    "\n",
    ">The following has been extracted from [section 4.1](http://www.nltk.org/book/ch04.html#back-to-the-basics) of Bird et al. (2009)\n",
    "\n",
    "Assignment always **copies the value of an expression**. The problem: what is a value? We've already said that the value is a memory location, but this has **different consequences on different kinds of data types**. \n",
    "\n",
    "<br>\n",
    "\n",
    "**This is a frequent source of errors in everyone's programs!</font>**\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"hello!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Case 1: Simple types & copy**\n",
    "\n",
    "When copying **simple data types** like numbers of strings nothing counter-intuitive happens. \n",
    "\n",
    "In what follows, what is copied from `foo` to `bar` is the string \"Monty\". As a consequence, when `foo` is changed, `bar` in unaffected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Monty'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'Python'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# copy & original value\n",
    "foo = \"Monty\"\n",
    "bar = foo\n",
    "foo = \"Python\"\n",
    "bar\n",
    "foo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Case 2: Complex types & reference**\n",
    "\n",
    "However, the value of a structured (or compound, or complex) object such as a list is actually **just a reference to the object**. \n",
    "\n",
    "When a list is copied, what is actually copied not the content of the variable, but only its object reference. \n",
    "\n",
    "As a consequence, **updating one will affect the other**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Bodkin', 'Python']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "['Bodkin', 'Python']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mm... what?\n",
    "foo = [\"Monty\", \"Python\"]\n",
    "bar = foo\n",
    "foo[0] = \"Bodkin\"\n",
    "bar\n",
    "foo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A list `foo` is a reference to an object stored at location 3133 (which is itself a series of pointers to other locations holding strings). \n",
    "\n",
    "When we assign `bar = foo`, it is just the object reference 3133 that gets copied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/array-memory.png)\n",
    "\n",
    "\n",
    "Source: [Bird et al. (2009)](http://www.nltk.org/book/ch04.html#back-to-the-basics)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the function `id()` to show that\n",
    "\n",
    "1. even if the memory location of the objects inside the list is changed\n",
    "\n",
    "2. the memory location of the two lists are left unchanged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "foo = [\"Monty\", \"Python\"]\n",
    "bar = foo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4453025024\n",
      "4453025024\n"
     ]
    }
   ],
   "source": [
    "print(id(bar))\n",
    "print(id(foo))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "foo[0] = \"Bodkin\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4453025024\n",
      "4453025024\n"
     ]
    }
   ],
   "source": [
    "print(id(bar))\n",
    "print(id(foo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with lists of lists can help to clarify the issue. \n",
    "\n",
    "Let's create a list of lists, in which the sublists are copies of each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[], [], []]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty = []\n",
    "nested = [empty, empty, empty]\n",
    "nested"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Python'], ['Python'], ['Python']]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the empty list might look like a primitive objects, but...\n",
    "nested[1].append('Python')\n",
    "nested"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing one of the items inside our nested list of lists changed them all. \n",
    "\n",
    "This is because each of the three elements is actually just a reference to one and the same list in memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Note that if we **modify the reference**, i.e. change which lists are in `nested`, the other sublists are unaffected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Python'], ['Monty'], ['Python']]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nested[1] = [\"Monty\"]\n",
    "nested"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A solution to all these issues is to use the method `copy.deepcopy()` from the module `copy`, that would copy the structure but not the object references:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the module copy\n",
    "import copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# deep copy\n",
    "foo = [\"Monty\", \"Python\"]\n",
    "bar = copy.deepcopy(foo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4452710016\n",
      "4452708800\n"
     ]
    }
   ],
   "source": [
    "print(id(bar))\n",
    "print(id(foo))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Monty', 'Python']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see... yes, much better (depending on your goals, of course)\n",
    "foo[0] = \"Bodkin\"\n",
    "bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4452710016\n",
      "4452708800\n"
     ]
    }
   ],
   "source": [
    "print(id(bar))\n",
    "print(id(foo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions\n",
    "\n",
    "Functions are constructs that allows us to organize portions of code more than once in a program. \n",
    "\n",
    "The alternative way to obtain the same results without functions would be to copy the same portion of code every time it is needed. \n",
    "\n",
    "Functions in Python are defined by a `def` statement, following this template:\n",
    "\n",
    "```python\n",
    "def function_name(parameters):\n",
    "    \"\"\"\n",
    "    docstring\n",
    "    \"\"\"\n",
    "    function_body\n",
    "    return result\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The list of the parameters required by the function is reported between round brackets right after the name of the function. Each function may have **zero or more** parameters. When a function is called, its parameters are called **arguments**.\n",
    ">\n",
    "> The (optional) documentation string should be placed immediately after the function definition. There are many way to format your **docstring**, [PEP 287](https://www.python.org/dev/peps/pep-0257/) recommends reStructuredText, but more formats are available. See [this tutorial](http://daouzli.com/blog/docstring.html) for an introduction to the topic.\n",
    ">\n",
    "> The **indented** function body contains all the statements that are executed every time the function is called. When a `return` statement is executed, the function exits and its output is the argument of the `return` statement. \n",
    ">\n",
    "> When there is no return statement in the body function, or when a return statement with no arguments is executed, the function  returns `None`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, the following function calculates the number of characters in a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chars(s):\n",
    "    \"\"\"\n",
    "    Calculates the number of characters in a string\n",
    "    \"\"\"\n",
    "    if not type(s) is str:\n",
    "        return \"This is not a string!\"\n",
    "    r = len(s)\n",
    "    return r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The docstring is saved into a  `__doc__` variable and can be accessed by using the `help()` function or the IPython `?`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Calculates the number of characters in a string\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "# don't use this, it is just to make the point\n",
    "print(chars.__doc__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function chars in module __main__:\n",
      "\n",
      "chars(s)\n",
      "    Calculates the number of characters in a string\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# use one of this two\n",
    "help(chars)\n",
    "chars?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to execute the code included in a function, you have to **call the function**, either in your script or in the interactive shell. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars(\"voodoo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is not a string!'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars(1979)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
    "A function can receive any number of parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def higher(n1, n2, n3):\n",
    "    \"\"\"\n",
    "    find the higher of three numbers\n",
    "    \"\"\"\n",
    "    if n1 > n2 and n2 >= n3:\n",
    "        return n1\n",
    "    if n2 >= n3:\n",
    "        return n2\n",
    "    else:\n",
    "        return n3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a parameter can be passed either by position\n",
    "higher(4, 2, 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or by name\n",
    "higher(n3 = 8, n1 = 4, n2 = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optional Parameters\n",
    "\n",
    "In some situation it may be useful to have a default parameter value, that is used when a call leaves an arguments **unspecified**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def higher(n1, n2 = 0, n3 = 0):\n",
    "    \"\"\"\n",
    "    find the higher of three numbers\n",
    "    \"\"\"\n",
    "    if n1 > n2 and n2 >= n3:\n",
    "        return n1\n",
    "    if n2 >= n3:\n",
    "        return n2\n",
    "    else:\n",
    "        return n3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "higher(9,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "higher(-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arbitrary Number of Parameters\n",
    "\n",
    "A different situation is when we want our function to have an unspecified number of parameters. Python functions admit the so-called \"tuple references\", marked by an asterisk `*` in front of the last parameter  (that becomes a tuple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_params(*params):\n",
    "    print (\"your input:\")\n",
    "    print (params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "your input:\n",
      "('Down from my ceiling', 'Drips great noise', 'It drips on my head through a hole in the roof')\n"
     ]
    }
   ],
   "source": [
    "print_params(\"Down from my ceiling\", \"Drips great noise\", \"It drips on my head through a hole in the roof\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A Note on Parameter Passing\n",
    "\n",
    "The way in which variable works in Python has a great influence on how they are passed to functions, and how they are affected by it. \n",
    "\n",
    "The following function, for instance, accepts two parameteres, a string and a list. \n",
    "\n",
    "But while the string is left unaffected by the function, the list is changed. Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reference(another_string, another_list):\n",
    "    another_string = \"new string value\"\n",
    "    another_list.append(\"new list value\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "old string value\n",
      "['old list value', 'new list value']\n"
     ]
    }
   ],
   "source": [
    "a_string = \"old string value\"\n",
    "a_list = [\"old list value\"]\n",
    "\n",
    "reference(a_string, a_list)    \n",
    "    \n",
    "print (a_string)\n",
    "print (a_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The effects of our function on the two variables can be understood by recalling how assignment works in Python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'old string value'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is what happened to the variable a_string\n",
    "a_string = \"old string value\"\n",
    "another_string = a_string\n",
    "another_string = \"new string value\"\n",
    "a_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['old list value', 'new list value']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is what happened to the variable a_list\n",
    "a_list = [\"old list value\"]\n",
    "another_list = a_list\n",
    "another_list.append(\"new list value\")\n",
    "a_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables Scope\n",
    "\n",
    "The scope of a variable determines where in the program it can be accessed, and depends on where it has been declared.\n",
    "\n",
    "**Variables declared inside a function are visible only inside the function itself**. That, they can be used only by the code inside that function. \n",
    "\n",
    "Global functions, on the other side, are those declared outside a function body. \n",
    "\n",
    "When looking for the reference of an object, the Python interpreter follows the **LGB rule**:\n",
    "\n",
    "- LOCAL: first looks in the names locally defined by the function\n",
    "\n",
    "\n",
    "- GLOBAL: if nothing is found, looks in the names defined globally in the module\n",
    "\n",
    "\n",
    "- BUILT-IN: if nothing is found, check if the name is a Python built-in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(inside the bar) I'm 18\n",
      "(outside the bar) I'm 17\n"
     ]
    }
   ],
   "source": [
    "age = 17  # global variable\n",
    "\n",
    "def try_to_buy_a_beer():\n",
    "    age = 18  # local variable\n",
    "    print(\"(inside the bar) I'm \" + str(age))\n",
    "\n",
    "try_to_buy_a_beer()\n",
    "print(\"(outside the bar) I'm \" + str(age))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Extra) Recursive Functions\n",
    "\n",
    "A recursive function is a function that recalls itself in its body, usually returning the output of this function call. \n",
    "\n",
    "That is, a recursive function must satisfy two conditions:\n",
    "\n",
    "- it must have a state in which recursion terminates (**base case**) (one or more)\n",
    "\n",
    "\n",
    "- it must recall itself (**recursive call**) (one or more \"in parallel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Fibonacci sequence is a classic example of recursion: in it every number is found by adding up the two numbers before (apart from the first two member of the sequence, that are $0$ and $1$ by definition):\n",
    "\n",
    "$$\n",
    "f(n) = \\begin{cases}\n",
    "               0               & n = 0\\\\\n",
    "               1               & n = 1\\\\\n",
    "               f(n-1) + f(n-2) & \\text{otherwise}\n",
    "           \\end{cases}\n",
    "$$\n",
    "\n",
    "\n",
    "The following sequence of numbers is generated according to this rule:\n",
    "\n",
    "$$\\{\\;0,\\;1,\\;1,\\;2,\\;3,\\;5,\\;8,\\;13,\\;21,\\;34,\\;55,\\;89,\\;144,\\;233,\\;377,\\;610,\\;987,\\;1597,\\;2584,\\;4181,\\;6765\\;\\ldots\\;\\}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Fibonacci numbers by using a recursive functions that mimics the definition of the Fibonacci sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function has been called for n = 5\n",
      "- calling f for: 4 and 3\n",
      "function has been called for n = 4\n",
      "- calling f for: 3 and 2\n",
      "function has been called for n = 3\n",
      "- calling f for: 2 and 1\n",
      "function has been called for n = 2\n",
      "- calling f for: 1 and 0\n",
      "function has been called for n = 1\n",
      "function has been called for n = 0\n",
      ">> intermediate result: 1 + 0 = 1\n",
      "function has been called for n = 1\n",
      ">> intermediate result: 2 + 1 = 2\n",
      "function has been called for n = 2\n",
      "- calling f for: 1 and 0\n",
      "function has been called for n = 1\n",
      "function has been called for n = 0\n",
      ">> intermediate result: 1 + 0 = 1\n",
      ">> intermediate result: 3 + 2 = 3\n",
      "function has been called for n = 3\n",
      "- calling f for: 2 and 1\n",
      "function has been called for n = 2\n",
      "- calling f for: 1 and 0\n",
      "function has been called for n = 1\n",
      "function has been called for n = 0\n",
      ">> intermediate result: 1 + 0 = 1\n",
      "function has been called for n = 1\n",
      ">> intermediate result: 2 + 1 = 2\n",
      ">> intermediate result: 4 + 3 = 5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fibonacci_recursive_illustrated(n):\n",
    "    \"\"\"\n",
    "    recursive function to calculate a position in the Fibonacci sequence\n",
    "    \"\"\"\n",
    "    print (\"function has been called for n = \" + str(n))\n",
    "    if n == 0:\n",
    "        return 0\n",
    "    elif n == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        print (\"- calling f for: {x} and {y}\".format(x = n-1, y = n-2))\n",
    "        res = fibonacci_recursive_illustrated(n-1) + fibonacci_recursive_illustrated(n-2)\n",
    "        print (\">> intermediate result: {x} + {y} = {r}\".format(x = n-1, y = n-2, r = res))\n",
    "        return res\n",
    "\n",
    "fibonacci_recursive_illustrated(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/fibonacci_tree.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notwithstanding their simplicity, recursive functions can be **inefficient**. Indeed, they require information to be stacked every time a function is called, so that once the function is completed, execution continues from where it left off. \n",
    "\n",
    "Note, moreover, that our recursive implementation performs the same calculation many times. In our example, `fibonacci_recursive_illustrated(3)` is called 2 times, `fibonacci_recursive_illustrated(2)` 3 times and so forth. \n",
    "\n",
    "Taken together, this sounds pretty inefficient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking Python Code\n",
    "\n",
    "Testing the execution time of a function is quite straightforward using the `%timeit` and `%%timeit` IPython magics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test our recursive function to calculate a position in the Fibonacci sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's rewrite our function without the print statements\n",
    "def fibonacci_recursive(n):\n",
    "    \"\"\"\n",
    "    recursive function to calculate a position in the Fibonacci sequence\n",
    "    \"\"\"\n",
    "    if n == 0:\n",
    "        return 0\n",
    "    elif n == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        res = fibonacci_recursive(n-1) + fibonacci_recursive(n-2)\n",
    "        return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.78 ms ± 178 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit fibonacci_recursive(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This magic command execute a statement multiple times in order to get reliable estimates. The actual number of calls can be specified with the parameters:\n",
    "\n",
    "- `-n N`: execute the given statement N times in a run\n",
    "\n",
    "- `-r M`: run M loops and take the best result each time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.58 ms ± 72.8 µs per loop (mean ± std. dev. of 10 runs, 50 loops each)\n"
     ]
    }
   ],
   "source": [
    "# let's play with these options\n",
    "%timeit -n 50 -r 10 fibonacci_recursive(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test our hypothesis about the recursive functions: i.e. that they are inefficient and that an iterative solution would be way faster.\n",
    "\n",
    "Let's implement an iterative version of our fibonacci function and test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fibonacci_iterative(n):\n",
    "    \"\"\"\n",
    "    iterative function to calculate a position in the Fibonacci sequence\n",
    "    \"\"\"\n",
    "    n1 = 0\n",
    "    n2 = 1\n",
    "    for i in range(n):\n",
    "        n1, n2 = n2, n1 + n2\n",
    "    return n1\n",
    "\n",
    "fibonacci_iterative(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.19 µs ± 59.1 ns per loop (mean ± std. dev. of 10 runs, 50 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit -n 50 -r 10 fibonacci_iterative(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Are there any fixes for the efficiency problem of recursive functions?**\n",
    "\n",
    "Sure. Two relevant keywords here are: \n",
    "\n",
    "* Tail recursion\n",
    "* Dynamic programming\n",
    "\n",
    "For more, see **[Section 4.7](http://www.nltk.org/book/ch04.html)** of: S. Bird, S., E. Klein & W. Loper (2009). Natural Language Processing with Python. Analyzing Text with the Natural Language Toolkit, O'Reilly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz\n",
    "\n",
    "* Write a function that takes a string as input and returns a dictionary of tokens (sequences of characters separated by whitespace) as keys, and the number of times they occur as values. The `split()` method for string might be useful.\n",
    "* Write an alternative version using just comprehensions.\n",
    "* Time both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Extra) Profiling Python Code\n",
    "\n",
    "While being optimal for **benchmarking** (i.e. for comparing different portions of code), the `%timeit` magic isn't suitable for **profiling** (i.e. for investigating a program's behavior). Profiling is vital when you want to identify the bottlenecks of your code in order to speed it up.\n",
    "\n",
    "The `%prun` and `%%prun` IPython magics can be used to run code through the python code profiler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "a_dictionary = dict((key, None) for key in random.sample(range(6000000), 600000)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_with_dict(input_dictionary, n):\n",
    "    \"\"\"\n",
    "    useless code in which you delete all key[x] entries from a dictionary, where 0 < x < n \n",
    "    \"\"\"\n",
    "    for a_number in range(n):\n",
    "        if a_number in input_dictionary.keys():\n",
    "            del(input_dictionary[a_number])\n",
    "        else:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.73 µs ± 166 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit play_with_dict(a_dictionary, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the magic `%prun` to see if there may be a bottleneck somewhere:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " "
     ]
    }
   ],
   "source": [
    "# the option -s to manipulate how our results are ordered (for more info `%prun?`, as usual)\n",
    "%prun -s cumtime play_with_dict(a_dictionary, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Recommended Reading**\n",
    ">\n",
    "> We won't discuss optimization techniques in this course, but having a look at the [Performance Tips](https://wiki.python.org/moin/PythonSpeed/PerformanceTips) on `python.org` won't hurt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### (Extra) Debugging\n",
    "\n",
    "**(For now, just an overview of the commands. Will be used later.)**\n",
    "\n",
    "To debug some code after it crashed (**post-morted debugging**), execute the `%debug` magic to launch the interactive debugger.\n",
    "\n",
    "Among the [commands](https://docs.python.org/2.7/library/pdb.html#debugger-commands) recognized by the debugger, the most important are:\n",
    "\n",
    "- `w` : the current position in the stack, with the most recent frame at the bottom. An arrow indicates the current frame\n",
    "- `l` : shows source code. Without arguments, list 11 lines around the current line. With one argument, list 11 lines around at that line.\n",
    "- `u`/`d` : to move up/down in the stack (the stack is the list of all active functions in a given position)\n",
    "- `p varname` : print the value of `varname`\n",
    "- `a` : print the argument of this function\n",
    "- `q` : quit from the debugger\n",
    "- `h` : print help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling Exceptions in Python\n",
    "\n",
    "Exceptions are anomalous conditions that modify the intended flow of a program. \n",
    "\n",
    "When not foreseen by the programmer, exceptions usually cause the program to end abruptly.\n",
    "\n",
    "Error handling is the process of responding to such exceptional occurrences by saving the state of execution of the program and executing a specialized function of code, the so called  the **exception handler**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The simplest way to handle exception in Python is to rely on a `try-except` block, i.e. a block of code with the following form:\n",
    "\n",
    "```python\n",
    "try:\n",
    "    code_in_which_the_exception_may_occurr\n",
    "except [ExceptionName]:\n",
    "    code_executed_in_case_of_error:\n",
    "```\n",
    "\n",
    "see the documentation for the list of [built-in Python exceptions](https://docs.python.org/2/library/exceptions.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "ename": "ZeroDivisionError",
     "evalue": "division by zero",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mZeroDivisionError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-e8b06114a416>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m9\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mx\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero"
     ]
    }
   ],
   "source": [
    "# Avoid at all costs!\n",
    "(x, y) = (9, 0)\n",
    "\n",
    "x/y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "are you seriously trying to divide by zero?\n"
     ]
    }
   ],
   "source": [
    "# a common exception is the division by zero\n",
    "(x, y) = (9, 0)\n",
    "\n",
    "try:\n",
    "    x / y\n",
    "except ZeroDivisionError:\n",
    "    print (\"are you seriously trying to divide by zero?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error message from the interpreter can be captured with the following syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "division by zero\n"
     ]
    }
   ],
   "source": [
    "(x, y) = (9, 0)\n",
    "\n",
    "try:\n",
    "    x / y\n",
    "except ZeroDivisionError as e:\n",
    "    print (e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two optional clauses of the `try-except` block are:\n",
    "\n",
    "- `else`: a portion of code that is executed iif the `try` clause didn't raise any exception\n",
    "\n",
    "\n",
    "- `finally`: a portion of code that is executed under all circumstances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None,)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "are you seriously trying to divide by zero?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None,)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done!\n",
      "5 4\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None,)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the result is: 1.25\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None,)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done!\n"
     ]
    }
   ],
   "source": [
    "for x, y in [(9,0), (5,4)]:\n",
    "    print (x,y) ,\n",
    "    try:\n",
    "        res = x / y\n",
    "    except ZeroDivisionError:\n",
    "        print (\"are you seriously trying to divide by zero?\"),\n",
    "    else:\n",
    "        print (\"the result is: \" + str(res)),\n",
    "    finally: \n",
    "        print(\"done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classes\n",
    "\n",
    "A **class** is a user-programmed Python type.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Room:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = int()\n",
    "type(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "__main__.Room"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = Room()\n",
    "type(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We say that an **object** is an **instance** of a particular **class**.\n",
    "\n",
    "`__main__` is the name of the scope in which top-level code executes, where we've defined the class Room.\n",
    "\n",
    "We can add properties to objects directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "r.size = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'Room' object has no attribute 'size'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-57-2f27c41cfd3e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mr2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRoom\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mr2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'Room' object has no attribute 'size'"
     ]
    }
   ],
   "source": [
    "r2 = Room()\n",
    "r2.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add properties and functions to classes directly, in such a way that all objects of that type will have them. We do this via **constructors**.\n",
    "\n",
    "Classes are thus mostly used to express complex behaviour that many objects share."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Room(object):\n",
    "    def __init__(self, name, exits, capacity, occupants=[]):\n",
    "        self.name = name\n",
    "        self.occupants = occupants  # Note the default argument, occupants start empty\n",
    "        self.exits = exits\n",
    "        self.capacity = capacity\n",
    "\n",
    "    def overfull(self):\n",
    "        return len(self.occupants) > self.capacity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "r3 = Room(\"kitchen\",2,3)\n",
    "r4 = Room(\"bathroom\",1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "r4.occupants.append(\"Bill\")\n",
    "r4.occupants.append(\"Sue\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r4.overfull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object-oriented design\n",
    "\n",
    "In building a computer system to model a problem, therefore, we often want to make:\n",
    "\n",
    "* classes for each kind of thing in our system\n",
    "* methods for each capability of that kind\n",
    "* properties (defined in a constructor) for each piece of information describing that kind\n",
    "\n",
    "For more: https://alan-turing-institute.github.io/rsd-engineeringcourse/ch00python/101Classes.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz\n",
    "\n",
    "* Write a class representing a university course. It might model things such as:\n",
    "    - title, abstract, syllabus, room, time, ..\n",
    "    - enrolment capacity\n",
    "    - students and their grades?\n",
    "    - ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Input/Output\n",
    "\n",
    "A huge portion of our input data will come from files on disk, and a lot of our work will be saved in memory. So, mastering the art of reading and writing is crucial even in programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code opens a file in our filesystem, prints the first 10 lines and closes the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Hitch Hiker's Guide to the Galaxy \n",
      "\n",
      "\n",
      "\n",
      "for Jonny Brock and Clare Gorst \n",
      "\n",
      "and all other Arlingtoniansfor tea, sympathy, and a sofa\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Far out in the uncharted backwaters of the unfashionable  end  of\n",
      "\n",
      "the  western  spiral  arm  of  the Galaxy lies a small unregarded\n",
      "\n",
      "yellow sun.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "infile = open('data/adams-hhgttg.txt', 'r')\n",
    "for i, line in enumerate(infile):\n",
    "    if i == 10:\n",
    "        break\n",
    "    print(line)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key passage here is the one in which the `open()` function opens a file and return a **file object**, and it is commonly used with the following two parameters: the **name of the file** that we want to open and the **mode**. \n",
    "\n",
    "- **filename**: the name of the file to open\n",
    "\n",
    "- the **mode** in which we want to open a file: the most commonly used values are `r` for **reading** (default), `w` for **writing** (overwriting existing files), and `a` for **appending**. (Note that [the documentation](https://docs.python.org/2/library/functions.html#open) report mode values that may be necessary in some exceptional case)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**IMPORTANT**: every opened file should be **closed** by using the function `close()` before the end of the program, or the file could be unavailable to successive manipulations or for other programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other ways to read a text file, among which the use of the methods `read()` and `readlines()`, that would simplify the above function in:\n",
    "\n",
    "```python\n",
    "infile = open('data/adams-hhgttg.txt', 'rt')\n",
    "text = infile.read()\n",
    "print(text[:10])\n",
    "infile.close()\n",
    "```\n",
    "\n",
    "However, these methods **read the whole file at once**, thus creating huge problems when working with big corpora.\n",
    "\n",
    "In the solution we adopt here the input file is read line by line, so that at any given moment **only one line of text** is loaded into memory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing an output file in Python has a structure that is close to that we're ued in our reading examples above. The main difference are \n",
    "\n",
    "- the specification of the **mode** `w`\n",
    "\n",
    "\n",
    "- the use of the function `write()` for each line of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outfile = open('stuff/output-test-1.txt', 'w')\n",
    "outfile.write(\"My name is:\")\n",
    "outfile.write(\"John\")\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "> When writing line by line, it's up to you to take care of the **newlines** by appending `\\n` to each line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outfile = open('stuff/output-test-2.txt', 'w')\n",
    "outfile.write(\"My name is:\\n\")\n",
    "outfile.write(\"Alexander\")\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the With statement \n",
    "\n",
    "A `with` statement is used to wrap the execution of a block with methods defined by a **context manager** (where a context manager is a class that implements `__enter__` and `__exit__` methods). A statement of this sort follows the template:\n",
    "\n",
    "```python\n",
    "with controlled_execution() as variable_name:\n",
    "    code\n",
    "```\n",
    "\n",
    "where `controlled_execution()` is an arbitrary expression, `variable_name` is a single assignment target and `code` is the encapsulated code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the Python interpreter does when it meets one of such statement is:\n",
    "\n",
    "- it calls the `__enter__` method on the resulting value\n",
    "\n",
    "\n",
    "- it assigns whatever `__enter__` returns to the variable given by `as`\n",
    "\n",
    "\n",
    "- it executes the code in the body and...\n",
    "\n",
    "\n",
    "- ... **no matter what happens** call the `__exit__` method on the resulting value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fact that\n",
    "\n",
    "- the `open` statement is a **context manager**\n",
    "\n",
    "\n",
    "- file objects have  `__enter__` and `__exit__` methods, so that\n",
    "\n",
    "allows us to **open a file** adopting the following syntax:\n",
    "\n",
    "```python\n",
    "with open(file_name, mode) as opened_file_name:\n",
    "    code\n",
    "```\n",
    "\n",
    "that will open the file, keep it open as long as the code is execute and automaticlly close it as soon as the interpreter stops executing the nested code, **no matter what happens**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this construction to open files has three major advantages:\n",
    "\n",
    "- there is no need to explicitly  close the file (the file is automatically closed as soon as the nested code exits)\n",
    "\n",
    "\n",
    "- the file is closed automatically even when unhandled errors cause the program to crash\n",
    "\n",
    "\n",
    "- the code is way clearer (it is trivial to identify where in the code a file is opened ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b\"The Hitch Hiker's Guide to the Galaxy \\n\"\n",
      "b'\\n'\n",
      "b'for Jonny Brock and Clare Gorst \\n'\n",
      "b'and all other Arlingtoniansfor tea, sympathy, and a sofa\\n'\n",
      "b'\\n'\n",
      "b'\\n'\n",
      "b'\\n'\n",
      "b'Far out in the uncharted backwaters of the unfashionable  end  of\\n'\n",
      "b'the  western  spiral  arm  of  the Galaxy lies a small unregarded\\n'\n",
      "b'yellow sun.\\n'\n"
     ]
    }
   ],
   "source": [
    "# that's how I usually open files\n",
    "with open(\"data/adams-hhgttg.txt\", \"rb\") as infile:\n",
    "    for i, line in enumerate(infile):\n",
    "        if i == 10:\n",
    "            break\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Next time:\n",
    "\n",
    "- Read **[Sections 4.6, 4.8](http://www.nltk.org/book/ch04.html)** of: S. Bird, S., E. Klein & W. Loper (2009). Natural Language Processing with Python. Analyzing Text with the Natural Language Toolkit, O'Reilly.\n",
    "- Do the exercises below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1.\n",
    "\n",
    "Use the function `id()` to explain what happens to the variable `pizza` in the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['base', 'ham']\n"
     ]
    }
   ],
   "source": [
    "def eat(food):\n",
    "    food.append(\"ham\")\n",
    "    food = [\"pasta\", \"sugo\"]\n",
    "\n",
    "pizza = [\"base\"]\n",
    "eat(pizza)\n",
    "print(pizza)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.\n",
    "\n",
    "The [factorial](https://en.wikipedia.org/wiki/Factorial) of an integer $n$, defined as:\n",
    "\n",
    "$$\n",
    "n! = \\begin{cases}\n",
    "               1               & n = 1\\\\\n",
    "               n * (n-1)! & \\text{n > 1}\n",
    "           \\end{cases}\n",
    "$$\n",
    "\n",
    "is the product of all positive integers less than or equal to $n$. For example:\n",
    "\n",
    "$$4! = 4 * 3 * 2 * 1$$\n",
    "\n",
    "$$3! = 3 * 2 * 1$$\n",
    "\n",
    "The factorial operation can be implemented in Python both as a recursive function and as an iterative functions. \n",
    "\n",
    "Write an example of both and benchmark them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3.\n",
    "\n",
    "Read the file `data/adams-hhgttg.txt` and:\n",
    "\n",
    "- Count the number of lines in the file\n",
    "\n",
    "- Count the number of non-empty lines\n",
    "\n",
    "- Read each line of the input file, remove its newline character and write it to file `stuff/adams-output.txt`\n",
    "\n",
    "- Compute the average number of alphanumeric characters per line\n",
    "\n",
    "- Identify all the unique words used in the text (no duplicates!) and write them in a text file called `stuff/lexicon.txt` (one word per line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}