04_presenting/web_scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a simple example of webscraping, we'd like to make a small library of images downloaded from Bing.\n",
    "\n",
    "**Note** Bing has changed its code since this notebook, so the scraping does not work anymore. The general idea remains the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Adjust the path to your repo\n",
    "path_to_repo = '/Users/ruben/repo/personal/ga/DAT-23-NYC/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "query = 'cats with hats'  # only use normal characters (azAZ10) and spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Download the webpage\n",
    "- mine the html code\n",
    "- find the relevant images\n",
    "- download the images to disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Do-It-Yourself Way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The package `requests` let you make HTTP requests like GET and POST.\n",
    "- To install the package, simply type in your terminal:\n",
    "\n",
    "    ```pip install requests```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "url = 'http://www.bing.com/images/search?q=' + query.replace(' ', '\\%20')\n",
    "req = requests.get(url)\n",
    "html = req.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html><html lang=\"en\" xml:lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\" xmlns:Web=\"http://schemas.live.com/Web/\"><head><meta content=\"text/html; charset=utf-8\" http-equiv=\"content-type\" /><script type=\"text/javascript\">//<![CDATA[\n",
      "si_ST=new Date\n",
      "//]]></script><script type=\"text/javascript\"\n"
     ]
    }
   ],
   "source": [
    "print html[:300]  # print first characters from html page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Have a look at how the html looks, and try to parse the file yourself.\n",
    "- We noticed that the search results are always of the form `<img class=\"img_hid\" src2=\"https://tse4.mm.bing.net/th?id=JN.8aaYIEhuqMgwcBHAyaTh2A&amp;w=221&amp;h=172&amp;c=7&amp;rs=1&amp;qlt=90&amp;o=4&amp;pid=1.1\" style=\"width:221px;height:172px;\" width=\"221\" height=\"172\"/>`\n",
    "- So let's look for the pattern `<img class=\"img_hid\" src2=\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 35 links\n"
     ]
    }
   ],
   "source": [
    "q = '<img class=\"img_hid\" src2=\"'\n",
    "\n",
    "urls = []\n",
    "pos, end = 0, 0\n",
    "while html[end:].find(q) >= 0:\n",
    "    pos = pos + html[pos:].find(q) + len(q)  # go to position of image url\n",
    "    end = pos + html[pos:].find('\"')  # look for end of image url\n",
    "    urls.append(html[pos:end])\n",
    "print \"Found\", len(urls), \"links\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating folder /Users/ruben/repo/personal/ga/DAT-23-NYC/data/images/cats_with_hats/\n"
     ]
    }
   ],
   "source": [
    "# create folder and subfolders for the image library\n",
    "import os\n",
    "path_to_imgs = path_to_repo + 'data/images/' + query.replace(' ', '_') + '/'\n",
    "try:\n",
    "    os.makedirs(path_to_imgs)  \n",
    "except OSError as e:\n",
    "    if e.errno != 17:  # if folder already exists, no problem (just overwrite)\n",
    "        raise e  # otherwise, raise e again\n",
    "print \"Creating folder\", path_to_imgs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 35 images!\n"
     ]
    }
   ],
   "source": [
    "for no, url in enumerate(urls):\n",
    "    req = requests.get(url)\n",
    "    save_as_file = path_to_imgs + '%03d' % no + '.jpeg'\n",
    "    with open(save_as_file, 'wb') as f:\n",
    "        f.write(req.content)\n",
    "print \"Saved\", len(urls), \"images!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if we indeed have an appropriate picture downloaded to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsK\nCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQU\nFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCACkASMDASIA\nAhEBAxEB/8QAHQAAAAcBAQEAAAAAAAAAAAAAAAMEBQYHCAIBCf/EAEEQAAIBAwIEBAQDBwIDCAMA\nAAECAwAEEQUhBhIxQQcTUWEUInGBCDKRFSNCUqGxwTPRJENyFhdic4Lh8PE0U2P/xAAcAQACAwEB\nAQEAAAAAAAAAAAABAgADBAUHBgj/xAAyEQACAgEDAgQFAwMFAQAAAAAAAQIRAwQhMRJBBSJRYQYT\nMoGxBxRxocHxFheCkdHw/9oADAMBAAIRAxEAPwDAasckZo9WJQ7mkks6WyFnIAG5JNMuocYoislq\ngkP8x6VTxsX026JSj8uOZsfeibnXbSzHz3SqR1A3P6VXt1rF5cghp25T/CO1IOTJJbc+tTkdQZYM\nvHtlDtGsk47MBj+9I5fEiUf6MAC+rNvUMHYdq65R6betCkPGGxJZOPdTZ/kZE9zmip+MtYkJBuh9\nQKj53XHejBuPapSLFBC+TiPU5T+8u5G9MnpSZ7+5n2a4cj/qNFgcwr0KFIOKXYZYwGVhseZ/cnNc\nUZjIJG1FmimO4nP3r0LzV7yUaijl5eh65o3W4VFhaLynOM0qs9MuNSlIgieUDqEGTinLQ9Fi1CNp\nZZGVFIwqjdx9an2lta2ViojBjjOxCDlJwM4z36ismXULHtFWz6vwrwGXiFTzTUYv/tkRs/DTVbpQ\n4iWFGJzzOPl9jSz/ALrr5JliF1Arddz7VPrUxIHjW5KTu6qXkbbBGR/frTnYWlzdagkdujX/ACpz\nKY4yzP2w3Lnrnauf+6zTlt+D1DH8H+DxhHrbb9epf2ory08Nb4ggX0KP0HKDufTNGNwjxHY2xe1u\n/iW5iPLVvTqd6tNOHb8pLOmlXckJl8vmNs+UIHbbtmkcMsKWywxkwNgsW6MM9QaLz6iG81/Q3r4N\n8C1EXCEqdcqV/nYqqbiTVNJlVL2I4GQyuCpBpz0/jS2uD8xaBj/Oas2ewsr0j4q3inYjH70ZJ96h\nHFnhYtxGbnTIWikXfyGbIb6Vfi1cZOpKj5Dxf9O9XpMbzaOfzUuyXmr8P7Htvf8An5YPzqemDRhc\n55skVWSXl9oVxgsyMDvEeh9ql2i8TRakgjb5Z+6Hv9K6OzPIJwlB1Jbj+GPqaBY5G5ooSHOCa7Pz\nDbtQ7lNbhYY+Ydz1ryRiT1PSvOb5+n3rmQnNFkYUWPqa8diR1NevtXLbqKgu4SzHOxIPr2pO5Pzb\nnNKCMg5opx1okvuJJWIiG5pOSdtzSqb/AE/QCkh6rQEtCaQ7nekkpORvSqUbnIpPIFOcnBooF7id\njuaFGBW/lz70KNoPUiPahqkt/IS5wP5QdqRBuU7CvTXJ60VZdsuDtXwucd67OCcUUPy496MB5mNA\nZAx+o6iveavSO/c9a85aA50PzZ9uldqABvtXONqMTIXagWJABA6V6Dk717gtXqpvQ2LUmeMuScHb\nFeRxhjR9vaz3XMsUMkjDsiFj+gotQ0MnI6Mj9eVhg/pR7A2s9xjau4U8yRVA3zXHU4pVYjE4G2fe\nkbpWX449U0mSqwijihU8pbBICY2B/wDupIHP/AWQiAMcXmliMEc24qN29z5SqmNj7VJCJH1OeByE\nMaJEjHYZVR3rj5O9nsnhrSlGMfx/T8E08OeCrzxF4ks9PtLWOORiqyzznCJv1bPt3reHhj4XaD4c\nW0Om6JxBp9pdqzRvO6xmS6lHzfLzYKp1A69qzB+G7heefg3X9bmlMckV0kTTRSYYoBkBR3GTuaiV\n/JrnH/FF6LJZ7u9jkd4oi5LKiZOAR02H6mu1ha0OKORQuU/x7HPz4snxNq82leb5eLCm26Vt+/8A\nk+oM/E2nW3B015q19Y6BJJtFdyoipI4yBzZ679fWs+fip/DM3GOlWXHfAGj2dxcvbtNqdhp2MXGF\n5jPCvQkgHKjfYGqb0O/4+/FDHp2lyKtlo3DFlIpuJlPLJKFJOV2JY4C+g3q1vwPeN9zewXHAWrXE\nvJbM13psw/1YihPmRAn+HBzjtvWnJqI6mSx5I+SXD9znaTwfU+A6efiOg1KlnxW8mPldD9+726vb\nn+cYadqz3F80DQqkgPKDPn5Vxmlcss0i5kT5F3DKcsTVmfivPDjeOPFcmgQfC6fJIiNHEOVUn5AZ\nCANgOcscfWqtt1kkbMd5zsBtnofpXyuWHycjguzo/QfhGrlrtJjzyTXWk62tX2ZUfilpYtOI1mVc\nLcr5mCMEEbHNQuOaS3k5oyUYHqDVreLti01laXLKGnTMTSJ0I7feqsEeY8+ldjTy6scbPzN8X6L9\np4znglSb6l/yV/keoeM7qMKrRI2O+d6X23GxllCtAVHXOc7VF7e0mvLiKKCN5ZJGCqiqSWJ6AAVp\nXwt/DFbtZJf8UCS7unHN+yreURpEO3nSd/8ApWtqjfB8JJJFQWnFlpdEYcDPTmGKcjdI5HK6tn0O\na1pHwpoXD+mPa23DWjxWYADwG0UiXfoXbf75/wBqyL44aFDwP4lX8GksYrCdI7q3XI2RxnH2OR9q\nLxtdyjl0Hu2TiuWfY7bCo9pHE6T4juGCydAcdafBICnXIb0qqicHR3FFSryjO9GA9BRUzZWorFYm\nkXMe9JCu49qWOP3WKSt8i5NGxHzsJZVztSSXK5GAfelkm4pO4BG9KBbCbLdqFGcvtQqWN1L1Ijz7\n9N64IJOcbV0RvQGatLeTwDFdJ3rw10nelHR3QwaA611ULUgDHrRidKHlZroDG1LaZYkeiugM5+le\nEUAvNkZxt1pbLlsam8OdSi4e0W3sbdWgYQqW8khHyRknm699qm9/+x+NLEWeuaZBq4KciNdwK0o9\n/MXDqffNUVwpxHJNp2nvzM7vGEwOoKgAj17A/erG0vWUhKNeCR2JHyod1+q9RWtNUcxp2yqPEXwe\nTQpJbzh83Fza5LNYXHzTwjuQ3/MUfqO+etVhASGdt8hTj7VrDU3F4iz22VuEICkSZI96pjjvhNRq\njXdtCIpJcieJRgMSDllH6ZFVZILmJsw5JJpDEEJuY48bFVX7EDenW/aT9o3488o4mYJk7DfFNQdy\n1nNggeRGxx9AD/alWtQSW97e83M6mZ/nxuPmPauPKO9Hq+ky+RTq9m/wbH8DL2z1bwYsLK2fkkSR\n7W6HRecnIyfTfO3rSPw08K+INE8b1mt1j/Zkdwx88PgMrZPLjue2+3Q1nXwm8Vr3w41q35jLPo8k\nitcW6jPN7gdzWpLfxs4Z0mC7vBqcstncnmhkiP72JyNg0eObbuK7cMmHPjh8x04/2ONgjq9BmzvT\nK1lTT/h//M1UIbSD41LOyhs7SY+VLKq8oLFcHPL39+9VD4R/h/sfDvxOl4og4rtoLS3eWWO1kTJS\nJlPOXcnHL169j60i4Y/FTwnfLLZajdtdQPGoMjWpUEj2G4+tVV45+NEXF9hc6JwvpcllYTYF1dzn\n57ggghUB3VDgVbqNVpkut03Hj+Tf4P4L4zJzw4oyjHKqk62rvz7dkVl4x8YWPHviVxJq7QQWqXd4\nzxCzOECA8oxnffHN96jdtDZhECN5bqDk83XfbNEJYLcTmWXTzFcqm4J2Jz/WnCCFlhTms0BeQK55\nd19Pp1NfHzm5zc3yz3zQaX5GGEFFUlS2fC+2xHvEKGYcMyMhWZAwYkdVxVQNbPd3cNvbQySyzuES\nJF5mZz0UAVcviFaNHpIhsFkKzNh8fMWPbH37VZngT4HjhSFNf1aIyavIAYkkXItlO+xH8Z7n+Ebd\nTXY0cW4Hhf6gyUvFqqqirFvgN4DQ8JWMOp6jEs+v3C5kJwwtF/8A1p/4sdW+wq8IdFtBG1qT8OQQ\nxYrsfUjHSl2i6dI7qpjdTKccueUAf+1FcYRnSLC6mc8gVgnO/TJH9s5613IQilZ5Fkk5SpEL43jG\nmWgOFCAHluYskde433x7VkH8TupfHeI9vEQTJaaZbwOcg/NhnIP05xWi+LNZvb64tbR4pPKa4SJ4\nyccu+/1wenqKx3x/qsmuca67eysWeW9lwW/lDEL+gAqmcr3QOjpaRHUODzDZvWpPw1rDzg282C6n\n5D3IqNeXSnTpvhb2KQDcHH2rOwtFgKSTvXDj5B60IXDxhgc5rs9KUq4dCZxsc7UkcEsRjb1pZN+U\n0mOxUUFuViOVSTvt7UQ4ODnNKZ/9X/NESdKgK9AjPvQrrAPvQptiUvRESoYxQPWvR0o7m6gLgsK9\nI+agNqA61OA0e4GKMjArjGTXY22FKMgUYv5aKJ+X3o2LegyxcnWK8Ow2rpRlj6V4cc4FIWk68O5i\n3NEdzbsJkJOMAjlJ+xAq6dDe1hVWWSG7lIHMxIH2wf8ANUJwReR2urwrJ86zKYzFyFuY9R/ar5sY\nGuNPScJ8EoXIBjCfbLZ/sa14GmjDn2kSGaWyjIKlgynHMGDIpPUA7DB96r3ju6hXziUMUrHmXA2+\no/SnpvJYHCJPcE4Vy7Of8Af/ADaonr9r5aTR3EYKhfMWNJF5l919PfH6VdLcohJwdkVnRDaQzKDI\niSGEkDGx+dD9MMf0p24hdjq5mWHzomjim5PUNGp2++aj1ok781rE4+Hl/cpKDtzZJhY+nzc0ZHbm\nHtXV1rs82nabctFP5sKNazR9CGQ5AP2cD7Vxc+mm22lZ6Z4T47pYwWLJLpr/AD+SY2upQm2iaOxV\np1ZSByY3+tPFvqbXKs0qRxSEEBi2P8GqwTimK6mihihunldQwXm29P77fanWfU+ILaWIJpiThpDG\nqtGT8w7Z+hH61iWizPhHo+H408N07XVlvbsr+72J/ZzXZk2ureRP4igyCfenFPLlumK3R5gBgEEI\nO23r9qpWbizidNQlFrH8OFkK8kceVJG53NSbifReM5eEU1K7vEit2jjkaOIAcinpkjpk5po+H5m7\ndD/7k+F4YNQhOTXHo/u5WvutvQs5pGLFJphK7jlBIwBnv65ruJhbwEvIWYSEFnABJx6e3+aqHhGz\n4g4p08C61G6iiSUeWi/LzADYg9B9d6mmscI6rczabp41ee6+DXmkmVTnLAMxwN2wpA+1WLw7J6oL\n/VHQcrBO+26+97/+37E74aFu2r288/lfD27+a7TAYBA2wO57j6VbPDfHVtqbqFQwQB/3cbEMNzn5\nj0J96zzo9pJJFI1yjG1QHl8zmJZjjPP35sZz2ycdBVmcIkXVnzrdx6daRIpaCOP8vvnb275+ldXT\nYXgjXc8g+J/iH/UGt/cqPTFKku/39zTPC+qWb3SxG5s158BuWbkdduwPXenDing614j0e/jhn+Jj\nnjEbsjDMZB2JXO+MnJ2qkrS+fT7ZJbKyupDyBvPkwHk9SR2G/U1P+BPF+OyiCavErRE8ySlx5iH1\n2657itql6nxDhN7xRSvidoN1wTMpvFDGyia4SRW+V0RCwYHG4yBWG5ZTMfMY80j/ADsT1JPU19WP\nHiy07ivwl1ee2NnrJFtKIo1IS4gQjLch+UnYHK/1FfK/UoktruQJzEZ/Kd8fQ9x71XlrZoaM3LaS\nEtdoBzr61zy13EPnGazlnCsm9n/ow7/wilLUltP/AMeI9+UUpdsLnvUM3ITJ+Ud6TSUpbcUmkIXr\nSkqxIep70TLRx60VL+tQTdhNCvQBjrQpbJsRLloZxXRHeuCKsRuZ6DmuumK5U5OO9dHtnb61GGO4\nAa7G9FqcsB1+lGtlfah7Dr1Aoy32o2IBW5SaKjYZo8KCc96RlqS5DFhQsOZm37KP966RlR8CGN//\nADATXIyCP713GNz3NJdFtbjro+py6VcJcKkSshyoEYYg/Q/71bOi8cScUrmIYaFQXGFLI3Q526fT\nFUqXYjfb3qxPDaNF026vIy6TRuEZz8wI2IHt36b1Zhb6qM+oilG0Ta7itoFW5EaXEwHOcKVJ32wV\nIIpnvdNhuIo9Qvo7u3tR8jO/7xMlsAE7EfoT7HFeX63PEN4JkhMzrlmkiIDgDoR0/Ke2M7UvT9p6\nnYSpZgKZAWaLLK7MDuAvYjGcrkHeugqOVZxpXApMckccT31heYT4i0YSBWIJ5uYYAKlRtsem1NNz\nBFpOqCS8kVpvionm8sARmRdmbB6pJG2T7q3pVreAU9jY6hO1xqS2l3O5BtmkCiflGdxjEi4/jUZB\n6kCqp8bNJvNG8Q7jSxG0KCbzocjGEJ5xt3U5yMbDf1pmktwdy9+Nfw5abo+oaLcaHYyCG6ljuRIF\n5sQl1cgdsjzC3vykVoTij8LL366SLGFYnitw78sfQ8vKTze7IB96nngh+zuIeFdElvY1mS3gQW7M\nScoVGP0zitIRSW8WkyKiKJAuzAbkdafylfW1ufNPjT8Nx0C/0bRktopivliZYwd2kcqzMeuAPT0q\nKeM/Bd1c61ofBNmogsnUG5lAHloqKDzBfTOdjW/eK+CzxSHuYyElaZS7IMMIuuAfXIG/pVO8S+HU\nVtxVrF845J47aG3hBGWAAOT92xn6UXTQVJ8GOL3SbPhPWn0yGNi1u0ac8zln/wDV+oOBUjAFraza\nrcK6NPMAzKg5Y+UnZCD2CgD7moqvCGqar4szadLymf4xpZJFY8p3OwJz2Gcdu/SrH8Q9Me1h0/R7\nGdY7S3YjH/MY9ThT23xzH2xVSLrPeF9ItNTS2W+KpGOeNol/eLcP8zLkDfIydu5xSi74euNLWyBg\nSCwYRNIGkySwB5lz3Pbb16mveEvkms4YkJWNi+csORyOXJOMk5J3OKsqbTbRrVVvXjRo2/eJO3+s\nwGOUZ6ZCqPXHben2YLrgQabNDcRqNVkmaNgF8i0APLj+EY69cd9+/eibZuHrmZ5pdLkiK48q1Eha\nZR2zgEZO3banCyjjsYHmMVvFynPIX5UAOdixxnbsOw9Kj3FfEsulIVtGa3E8SujLGiQgbfPnqxO2\nDuNqzZYvp8ro1YcqUvMUH+KvVL3RdS0VtP1K7t4JOeWCDm5XhHRs42OTt9qzhd3Ul9OZpjzSnOW6\ndd/81oD8VmqW93pPAVsZfO1NLe7uLh2YcwV5F5AQOn5GOcDORis9cpzVG6STLW4yfUgKpNGwxEyD\neuEUilFuhaVaDZXIltt8sMfsopQ/zKKTwD90n/TR52WgvczhLnbHeiH67jPvRztuaIkPv9qABK4x\nn09KJkOdhR8jZOKTNsetSwHgXahXokGOtCl3BuReOznDcgt5Sx7chqT6N4bapq0ayErbK4yBKNz9\nquS10+GMhmh5iD3G9OEDiScxopjAwTtXSWm33Zrba4K30TwRe5lHxFyzr3CDlBqfJ4P8P6TbB5rU\nTTHbDb1PNMjVIV8hSqjs3WlselfEy+dc5Y9o1PT3rVHBCPYO9bkGsfCDRLy1WSXT4Y0YdF2NJL38\nPGi6yrrYvLaMOrocgfXNWtb6PNOVaOAqn80rdftTzYaLJ0eQKvoNhTPDF8oS2uDNl5+G1YkZbfVp\nTID+Z49vpiojqfgzxDps/lxwpegnGYjj+lbA1S1s7GMKB58rHAOcYzTTNouFDIPnxvWeWlhLgdZJ\nR43MZavwnqugSAXtjNCW6ErkfrTZGMPjGCO1bOvOHvjF5Li2WdD1Vl2NQ/WfBPhy7SW4ksja3DH/\nAJDFc/as09FKriy6Od3bRmT+EfWn/QdaltLK5sFkKpKwcDGcEd6kXGvhc/C7fFRzh7XOArDDE1Et\nPhuxqcK2aeZPMeQSAbL2zis8cMoPzCajNGS6Yll8G29vqdtIstyzSRgYwHQYPUkg46+tPeo8UcG8\nNXNq1zPcypgFrWWRgC4O7/K2fp0B9TTNovhSwnEl3rzw6iRzIjAhGHQqcem3aorx74batwvxpYpq\nZkuFv4/Pt5Zgfn29/pWhy6Ec1rYvXgzxT8MtavEtpbVtOYvGInmTygZB0kDIcqffIPXqCRVv8SeH\nfDnHVzY6jA0N3dWIAWdmHRs/IcZDdcj/AOCsJcU8NX9vZqgiwzSqnICMnOcAD61NuHuPde8Jby0u\nLMu1o4HPaykkNy9cenTpRU9rEUlwzevhNrjaDY2llzcsMcoUDpgA9K1toOoDUNHFyDkMmDj0wKwL\n4d8UJxRo9lqUI5EuW8xVB/J33rZng1qLXmgPbs3MUOcexq9LYaRPeGdN8yzcsMtuCO3U1T3iLzLr\n8syBY4g2HBXdgp64/wCqtB2UHwNi8i4V+XmBPQHrvVDeJJt7HhbWdWuZh+7zMg68uT8y/Xv/APVA\nrjyZhs+E9I4Q4v1PWIJ3kurgswe4YM4Dkc3L/L6dMnNV74qcf8EcN68ive3dxdT5M7wS8zRHspky\nADtjAqB8S8e654pXt/BpRfTtLt52jdkcrJMT0YkHOOwqOcQcFTcK6Ro+oaVZ2t9f2OpQTLBcRiaF\n2yQoeM7MvMRzA+tURyJy6Q5MsYOiZ6V+IjQtI4jx+xbg2kjBUnu5mLMp3zzMD9ME960Hol9pHGGg\n2+raXPLbklvMPMkjAk9R2Hpn+lZb8ZtOl1iYwzSRPf3riK20+yj/AHUbMdlROwzsPb6VdvAfA+kc\nD6Ba6Ffw26XqgSSKWwBzLkg439aZT8zoaE1lh1cEsngjeAtPdRy8jcqyXM/NGenr0ByPbaqj4+M1\njcSPMJmRbjlJkjQc56jlc5Y4BxtgAelSzxB8PDwXZPxDwzPeW0EJXzLLzWZgGx8wJzyr69K6bRk4\n54XtWZoEwOdVcj5d8Etk77d8H+tW+w+xjvxIbUOIddvdYntJWDsFEpcyEKBgA+mBjAqE/wAVbeuv\nCS0/ZcsMZecuCZHC8oO35cZ3PT74rIHGvDbcK8R3mnlWXynOzdQDuM+9ZZxa3NGOfYY6VWYxKnpS\naKP3pdaR/P71S/QeTJJCcon0o5scnWksQKov0o5z8tCihsKeiJNzv/SjJPzb0RJ+lAAS+zfak77g\n0e+CKTSb0FsRbPY4zmhXFCiEv8g2tlLO7Z5MmnTh+wEVgl7dLnnw2W671GNZ1a0ytl8Qo52y5Y4A\nGakM/ENpHafBCePy1wAwbtsa+hTiaLTJlaafKikFSoO4b0FPViLayh55mBfoOY7moPqfibp+oW1j\nBZyCOVJAkvN8uQP705pf2t+yyJcRtJttzCmbTI5WS06nJLyjIiTsBSkT3FwFjh5ncjYDvTHpqJqr\nxr8Qkab8xDbipzp8cFnEiq4wMAOcAmilYKsb4tCa2KvKpL/m5j0FOFjohuGbJzgZLYoaprthpq+Z\ncXQLA48vOaYtR40/aTLHCwt4SAByEAnJ70+0eCcDpqFnFAXRbhefoBTPPoV7cnzOQXKgbBW3r22t\njecuJDIMDcnvTjBZXFnlop3i9QOlFb7B3Zmfxp1Z59TfTAvlrbf6g/8AF6VWPDmk3E9+8sEnlcg3\nGcZ96nHjXc+bx7q5aVU3XDBcBjj0qOcK311pwWdbc3Acg8rYAf2wK42S+t2ZGtyx+HeJ7y90ee2u\n4ne6t0J52B+YYxgtjbap9c6lZcV6dY6HxNJItpGyS6Zq3IrTWcgHY5wVO2UbG2ag9pf3d3DaDn+F\nDsCScK/Trv2270+W3DrT3Vutv8NcJNyliIypbPUAggY98UVUlQHsLb/w41nSLtru309NYtuf9zPp\nriQM2M7L+ZT7dvU1RniXoHEGn3fxF9pc9mkx+RZFIYgnc4rd3AngZxBfxRzx+bZ3HlrHz+YAoUHJ\nILHYEY6Del/HvgxBr3EdgdauIrm0iQicw55pCDkYJ9B19T0qlabpdpidaRX/AOHzhuXT/CvhyScc\nktzFzAb/ACoCQDv61sbwZi+D1e1YAtFIrISu9U5e2KPZqliBDCjLEoQAKnL2A9AB2q4/Ca5W3VIp\nCFCsSuOwPXb61qla2Low61ZoiBUmQgEcpHSso/jPtDw94Ua/HBG2JWVcIv5Vdhlq03p10lxEhjkJ\nbHMQKYPFvw7i8TOB77SpWjR5kwjSZxzds43pKvYqXllufKzwK4D4o40kutYsbSzitoYPhLjTZQyL\ndxLsDzb4bG4OKmnEPh3c6XfxPc6tpGlwCPmQzSszhMbEqoAJHpke9XfwtwpqvhpHrHDl/CbPU4Sw\nLQoQJ0PRlx+Zfftn2xWZ+OuHtZ1HUZbe5tpjEpLDzZUdXcEttjt2wwHes0sEZS6pcjPofCEemahw\n/wAFarPqmm3J4h4gZig1a8iJFuP/AOUeOVPqd/ep5we18HurwCa7vpDkXbozu7EZIOR0+ntvUI0L\ngWI6aW1JktWQqsk0d2rZyeblCg9v5Tvv0q5eBOArG9jln0bWL6IIo8ySedlIGOigKBnPp+taPYCp\nCy30PiN9MljuIGkt77JdHOAMsN9h642IPeq5spV4a1y8067RREku3l4RBjOQvcfQ+tXetvxJp0Fx\nILm0ksUXClnM8m+Oir33xg9M5qp/EjhibU7ia800N8QFUsXjKH3J996LW1gslenTpJMjcxSGTdQm\nMA+vp09O9ZW/F1w5p2mcYW13b+ZHcXEOZUdMMxBxzYrSHhFqE+t6Xd2s/LG9vIqAYI26nfp+tZs/\nF3qE7+JLWMty8sNvCrIrgEjm3O9LL6dxoclCCA522pbaxkKPX1rwLkZo+EkAHFYqL27HWIHkXPpX\nrkgda5jf5Bt2oO23SgVnDbg0nk60czZBoiVsmoxWJpGPSiH6UfJ70ncYB+lGrCnYWKFeLkjrQo9L\nDv6Fx250QM6gq2HYCNvmI7gfaurrW7fzGWwjgbP5xIvX3x2pHc6fHeyT3cKhWLYPyEYOe2aSwLaN\n888bPdIMcoBDZz0/v/UVstsppB1xb32pwqI9OjicIcyKMBl7/wBxXEsiaLbOqQsJAAQSSSCTRuL2\nKKOMm6h5ZCTG38Knt9Pb3prkZrVhI8hn59mRs4U56EUbG7bDxo0N7qV0Abhrcgcxw5BYdfWpDNZ3\nFsSkuvTyqAMJ5vKV9v7VXay6jdzs0SlIkTIRQcY9qdLXQLm5t0nnkcqyEnmc7j+E0VJgti3UNcgt\n4Jw1xcXExHKVMh2P/vUei4g1WVZI/iriHLA+YrZZTnYgHrj07ijtdurHSLIJGVmu2POSNwMf77VC\nNV4jubsFckKW5i2O5qKT9SKUi0bXxZ4k4ZuIwdTtp5ISOZZVID56H6Ed6mWgfigKYXVbSPlDgFoJ\nVI5T1O5H6VnJb9LqJodR8xoRulxGMvGfpkcy5/hz7j3KPDl25LWMsWpw45swSAv90OGH6ferI5px\n4ZZGclwWF4p6vF4icYT38Msa22yRcmxA98da40i0awMQbPOmOUgcuajen8PaxbwrJDpl3E+cZMZJ\n/QipPp/DHE+siOMWkp7cxVwR/TaqJNydsF3uWrwnpGi6uY5r9yZkIJjkkLYb3XO+fvitB+HGjWdr\newIFWCMbBBGApU/VQBWcuAfCfifSbyO6fzgpIPKEdwy+4K1p/hhHtLNTKrRhRuBG4A+2KMXvsCS2\nsuiTiCK20+KCJcKigZxy8xx/SoLxnqfLZC4UnKZAGcV7a3EV4UXzGkd9kCoTk0mv7ZbwFJAJlTbl\nFaChLuUtx349WHh7r1hbXsUl1FK5lEcDAEZHUk9NyBT5qn4wVs4rO90J4YrRo15op4svzZ+YZ6Ee\nhH1rOXjnoL8NcU3GpapbTTw8/LExXIRM9geuM0/8FaXpF0thK0Ec9uQHR405hJkZB9K5eXVfJe8W\nzv6bS/Pj5ZI2t4XfjKtOINZ4churJrCC9cwSMZPMCOThDkdj3zW0dMuBqNnFIxXDDPLncV8heH2s\n+J+N4OF+HtFntZ0lw2pQgIUkByCANyAe/TtX078F+HNQ4c4cij1TULrUbtwC8tycY2wFVeigelac\neVZFaTRi1eF4vqaf8Ex4v4J03i3T2hvbdJCPyS8o50Pse1Zm8RPBWeGdiEt2h5vk5I2I9sgYz99q\n13HIgxg746U0a1pMV7HzcmM9R3q+zm2fPseD1mrGDV7X4mIsW57KNgpBP5m+Y8pwCDjAFLr7hfwx\n4da3t7jiRdOiLBfLjvPz4GcFtyNtt8Vq3X/DqwvVYyWcYVDzIAcY+w/zWbfEvgXTIdMv7HS9PgtJ\npuYPcGD52Ps2DvnsaDQ6kmH6VxHod3btDw2jXFqi8izSP80gG2Qeh9ts0Tb8MwXETef8Q07sc/vW\ncD0BGDjfA32+lURwt4g8QeF2ox6drKi8tGbEbtAiuFzjoDn7Yq7IPH7hazYw3Vytreug+VupGM/o\nR2xURHfYrt9A1TgHiu/1CztHnspTieJfmkRe5Htt09qxt4y8bLx74g6pqPKI4xIYUUbYVdhkZ2rf\n+peKGjazDm2ntp52+RIDlUbvgtgFfU9q+efGut6BNxhqnxOiXNtdPcuXNpeLIpOevzL/AJpJq1Vj\nRkyNIoKHHbejYUwg75pzii0iSLntrbU58fm/eRqo+pxSeaezglURwOGYZwrhuX6tj+1ZunuP1M75\ngqgYIOOhrh2GKHmeZvj/AHrlxyr0pa2JZ4WHSk8m1HHGMGk8hHXNLyCwmR9qIYjBoyUg53ookY60\naFOBsMY/rQoA4HQH60Krt+g3VW1l0pxCkuqGzhcCOQ5WXHUEfKSPcUt03htk1G6kYZK4JflxkE7/\nAKU28I6c1vI9yViaNhhXPzfKPWn251qe7kSGHKhcIz5B26DmHX0H6V0EIzniDU55EeOJQ84Qq0pG\nMgjAGfUUw3vCdzcWyXiBmuXOG3yCevT6GpTb2jQX+LiBpFlBJfOeg9PcDr7V1d6nBYfDtAuY9iHP\n5XIGHHscY3oN+pPZETttMNnckzzFpEjyygYG3p7Yr3UNTN1CBapz+d+YRnptvtTXq+q/GalELWX5\n5SQS23Xt9/SpNpWmLpelyFo1Yhf9UDJxjv6j3pU7JRXl1wlcY8wDOfmx3xUZ1udZ5zapDgIBzbbf\nWrE1fXrSaBc+WsygDBzgDbv/AFqOX+tWcanFqk8YIOZDkt6jP+xooVuiEzQuVCAbAbH1pugeeznV\nkXnUNzBSucn2FWNZ2GkCM3F5ozWYf50e51BunqsQTmYH9Md6cJrvTbCyQ6dodzcDq08l8beJ89QE\niUMR9Xo7kUuwv8PIrrV4Wt7mJtMTHMrSzeUCD3CNg5+lWZwfwnqCXQNldahqcZbIFnaSMuAe7tsP\n0NQHgfjW70rVxd6do2g6eiYxPa2IlkPqDJKXcn3zWj9N1bU+JrKC4n1OaINgmGVuQH6KACfsKV78\nFibJLw7Y38Vsi/DW0C9D8VJlz9Qp/wAVKINI+KgYAwSsPzJFGVH64FROCyFtHGLq4lCDflkZhgfQ\n5Y/oKeNN1qytrxVjWd1G3ybAD1x3/UUrlQ1WS/TJbmxRYreGJ2LBMINlz708FpZHUNZBSjcrEd/e\nmfTP+NjZ7S8wvuvzKfU0+ec6iMiTmH/MOOtaE00VdLIn4neFFp4kcN3umyRKWlUqkvLkxn1rGXEO\nm8T+BOq2/DOpRtILeTntpLcYS4iIIyPevoFp8iKC6zO7+YML2PtWfvxTaho91DosNyY1uredSJwO\nZmUbkH0G4z9DVWTHHInaNWLNPE/I6LF/BVwFcvd3fG/ESRw3d6R8NFygcqYx1+lbjtb1hCnw+GGN\nvSse+D/E9nreh2phmMkUSIAsTYYOvtnGMVf2garIYeZLqVUY7iQflI9MDpVkIqKSiU5nKcnJlpW+\npXbN81r9+aneOa4lUDyQQevMarvTtZl+Utf4UfLlsEfepNa37MRi+X6LTNGbgP4it+WFlVGRnHr/\nAJ7VT/Edtf3M7xw2s8i7ggSr83sDmrW1ed7i3Mfmc2Rg/MRn6VFLnhwzHHxDkH+F/mWlZLM5eJ3h\nxPqUXzxtA2QUWeAMA3qCwzj6GqntPAaWaSKPUIrbyAT5ZMOCBk++f7Ct5WPDsUMPI0SuoB2BIX7L\n0pr1rgnTdQgP/CR+YuSuAFwffFCh1PsZP0vwpstGsZbq3tbZ544yvM/NydDsAScdu9fP/wAStLew\n4gvpYra0R2uHwbcI5Bz1+bO/0r6v+I/DkWlcIaj8R5ttmFiZVUPyj1GPT3r5ocf8GR3mqzGy1Sza\nOU82JJzEDvnPLIBvt2Y01eUN2yoRc3N7iO5mdim3Ixz/AO1d5aJwvlNy/wAw6U8ScF63A4aPTpbh\nQd3hIlB98oTS0aLMIwLi3eB8biRCv9xVFNElKhnh/Jvse1CXOPaltxYmNhy7qNs0jlB5fpVLjuMm\nnwJ3zyk+tJJKVSZ5M+vaksn5lzS1RLCJPaiyPloyTptRDHbrU3J32OeUGhXGCe9Cpb9Rut+podnb\nRNO+HgVHZV81miH5W6H7HpXnCNquoySXEpaPJC80YzuO36j+lMep3Dy3MtyriHnOJIzkDIG+f0/p\nUg0RprW0Z4/lhciQBeudjkfpWte4CR396BbpzRDzI1x+bod8j3B6/eqn1zihAbmyPIYWPNEnORy4\nzt9cf2qUcV8SxXCSOoKSuCeTdTGfX6Gq4i0ObV9QQoTIp+VsMBJt0JHY0juREqHjg+2hvVeVTLyM\n3N5TkFvqo7U8cQ8QzWFkOUrgDYDZj2zSqztrLQrTkdfLwuedSDn0371BNcni1i6laaaSS2TIjKAc\n8uOvL/udh7nahTewH6jParNq95cySlIo4iGct8oVT3Y9v89s07xXUU0Ijs4zcIrZS5KjKkbZUEHl\n+pyfTFRqe8N5cJAqs0KkCOGLJXPuepY//PSrA4a0VobMyXRisW5Q4WE8s2O+cH774p48UV87hnB/\nhJqHFd0L6WHz1LczyPOQxHqT1q4U8K7W2sUElvyIBgnmCpn0ydz9hSvwL1UGS8soreOT5g4uCwds\ne4I2+1W7ccNnUUZizg56Ekg1o6VVldtMqHhXgfS9JtHOnWwDlyZOT5Tnvlmy36BafLOO60q6drYr\nHzLk+Xkc3uxOWJ/9VS+LheWzPKFyP5lG4P09KM/YL3TB0QGVTgxkYyazzXoXRYz2epCZ+WclZMZP\nN3qRW8CGDIdj/NGjcv6/7UR/2b8yMuDkgDlYdRTctpc6ZdlI2ZQy7enuftWSTceTUqlwTHRpZrCX\nzkdliBxyL06b/WpkmvG8sgiR79QCMb1W9vrrciR8nIBucdh61JdH1TmmQIwbfHXFBZekbpDr5dd1\nSQxxv+zo84BjHMxGPWolq3hHDqEsAkYyzNkF3OSMDc5PTpVzaOYL8BJUCnHfcGnn/sta3a+YYxsC\nqoBsc9avU7JFpdig+E/C/UeGPNudGvZLQOegJ5Sc7c1XTwXrfFUAEeo20dzHkAsmVJqX6TwtCWwe\nbAk5hy++1Ta20ON1UGNQ57gYAq1SpbCSnF8oS8ORy3JHOmQdyGWp9pOnIkRPKAxHWkFjYLaIBjfF\nOUF4BEeXtsfY0FP1MzoPkg5V5Tg+9JzEC2Mb141y5zz8wX6ZpZbBGUN1NMpJitCUwyIQT0ro21vK\npbkKsR1G2aW3HyoM7GkM7tGFP8NWoqZB/E3SILrhu7gUuks6+TzLu2D1wO9Yz4h/BtpOtGctrDWd\nxzEmaC3RVLHs0P5c+pXl+la4451gajxLZaUqnmK84KncH6dMUw6hpF7GCx/4yPmIEkanJboAO+3v\nWyEVJboqbaexh7WvwDcRW5abTeIbOdGOQWBT+o6femFfwreKOhLlSLq1Xvb3hbI/6Wrd76LdWivJ\nHPIpRAnJLH6/m9j9KbZIXnkaONnifmy0bgqCo/lHb6VP28OwOtmBtc8KOMdIib4/Qbhkz1ks1dWP\nrkD/ADVaa7pM1kSJdLiTB6BHjP8ARsV9Obsajb25lik88Ic8mcnJ6HB7Urh4K0rifTVXWdJtLkuM\nZe3U8o7sciklpk+ALK0fJeVrF1LNZTx/+VPn+hWkzxacfmEt3D/1xq/9iK+ifHf4K+DeJ5JpdNtv\n2NI5ykmnEgZ9TG2Rj1xisy+Jv4OOM+C7eS80qMcTaemWb4RCs6KO5j6n6j9Kyz0812NEcsXyzO0l\nvApJFzzegaIr/bNI5UVSQrhvoCP7inG9sZbWZ4pUaOVDhkYYKn0INIZIsb9KyU73LvYTUK7CDH5q\nFLSGL51WCO7YyyIMqwIUdM8+P7UsiUadZLHFuipzKG35c4zj9aFCtAWQniSDkuWlSR0KMiAA7EEk\nYOetGPaRG9tZQoViVBC7Df2oUKV9iIcOJcG1hix8hY5x1IC5x9Nqqm6L3l9hpHRQwACHAA9qFClX\nYrkO1pAkFvqFtEPKEcm8ifnk2P5m79OmwqcaTplsuhLMYg7c4Qgk4O3U470KFXodfSWt4AaPZwa2\nwSEL5mS5znJ3rS0dqiQxhcgcooUKvj9JnlyO+n6Ha3ERZwxIGcg02a/w/aDS5Z0DxTIMiRGwaFCq\nZcDQ5POAoRqWkLNcMXkBYZOOxpk4v02G1S4uo+YSIFAB6btg7UKFc7JwbMfIlsNOgeJJeXDqvLke\nlJLi3FjcnyGeP5+XY9sGhQrDk4NcCX8FXE84VnnkYjbc1bukxl7YsZHyB60KFW4XsxJrcnuhadEQ\nclug7ipjYadF5an5s460KFbomOYL4GGQlWPTGDRGn2gdGzI529R60KFCf1CrgO8s+cic7FT64p70\n6xR7aNiz5Od80KFPj5BIVahpkUmnSuWcMg5gQd6Z7yHOiST8zeYqEjpjNChWyPJnZQthqdy/iDPC\n0pZArMueq5G+Kl2s6lNpkvlwFQoKJuucg9c0KFbsfBUxpmnnkZCJ3iBkHypgAZIz296iXiBrt7oE\nci283mKbhExOA+xxmhQq58CEu0awi1HQP2hLkTyEcwU4U4wOn3o2LTYheNGC4DHJIOCcjptQoU0S\nqYSqm2SMoxzICO230p0k06GaC4Zs5QALjttQoU7AyjvxD/h14I404f1LVLzTDb6rb2xmW+syscpI\nHRjghh9Qa+XWsWUdrezwLkojFQW67HFChXK1KSaZqwt2NMkaq2MenehQoVifJbJ+Zn//2Q==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename=path_to_imgs + '001.jpeg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The package `BeautifulSoup` will help you navigating through the HTML code\n",
    "- to install the package, simply type `pip install beautifulsoup4` \n",
    "- Documentation is available on their <a href=\"http://www.crummy.com/software/BeautifulSoup/\">website</a>\n",
    "- Especially have a look at their <a href=\"http://www.crummy.com/software/BeautifulSoup/bs4/doc/\">Quick Start</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of parsing the `html` page yourself, you feed it into `BeautifulSoup`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "soup = BeautifulSoup.BeautifulSoup(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `soup` object saves all the html page's attributes as class attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>cats\\ with\\ hats - Bing Images</title>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'title'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'cats\\\\ with\\\\ hats - Bing Images'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'head'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.parent.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common task is extracting all the URLs found within a page’s `<a>` tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/search?q=cats%5c+with%5c+hats&FORM=HDRSC1\n",
      "/images?FORM=HDRSC2\n",
      "/videos/search?q=cats%5c+with%5c+hats&FORM=HDRSC3\n",
      "/maps/default.aspx?q=cats%5c+with%5c+hats&mkt=en&FORM=HDRSC4\n",
      "/news/search?q=cats%5c+with%5c+hats&FORM=HDRSC6\n"
     ]
    }
   ],
   "source": [
    "# Let's limit to first 5\n",
    "for link in soup.findAll('a')[:5]:  \n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's look for all pictures.\n",
    "- We already saw earlier that the picture we're interested in are the ones with `<img>` tags with `src2` attributes, rather than `src`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 35 images\n"
     ]
    }
   ],
   "source": [
    "# Let's look for all pictures with src2 attributes\n",
    "urls = []\n",
    "for img in soup.findAll('img'):\n",
    "    if img.get('src2'):\n",
    "        urls.append(img.get('src2'))\n",
    "print \"Found\", len(urls), \"images\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating folder /Users/ruben/repo/personal/ga/DAT-23-NYC/data/images/cats_with_hats_bs/\n"
     ]
    }
   ],
   "source": [
    "# create folder and subfolders for the image library\n",
    "import os\n",
    "path_to_imgs = path_to_repo + 'data/images/' + query.replace(' ', '_') + '_bs/'  # add bs tag\n",
    "try:\n",
    "    os.makedirs(path_to_imgs)  \n",
    "except OSError as e:\n",
    "    if e.errno != 17:  # if folder already exists, no problem (just overwrite)\n",
    "        raise e  # otherwise, raise e again\n",
    "print \"Creating folder\", path_to_imgs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 35 images!\n"
     ]
    }
   ],
   "source": [
    "for no, url in enumerate(urls):\n",
    "    req = requests.get(url)\n",
    "    save_as_file = path_to_imgs + '%03d' % no + '.jpeg'\n",
    "    with open(save_as_file, 'wb') as f:\n",
    "        f.write(req.content)\n",
    "print \"Saved\", len(urls), \"images!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## import.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The website <a href=\"https://import.io/\">import.io</a> tries to automatize webscraping.\n",
    "- It doesn't work flawless, but it definitely gets you started if you quickly need some sample data\n",
    "- Have a look at the website and try to scrape another webpage. (Note that Bing might not be the best example to test out this engine.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Scrape your favorite site and save your valuable new-found data to disk!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- (**) Note that Bing is a dynamic website that automatically shows more pictures. We only found some small number of images, though. Is it possible to get a hundred or a thousand images? (Honestly, this can be quite a headache, and I don't know if it's easy to do.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}