Medium Stats Data Collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Medium Stats Data Collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal is to scrape baseline stats on stories from https://medium.com/me/stats in order to get a better understanding of how readers engage with a writers work. Note that this is a personal project and is in no way associated with Medium. Some general advice and implementations are adapted and revised from the following resources:\n",
    "* https://hackernoon.com/web-scraping-tutorial-with-python-tips-and-tricks-db070e70e071\n",
    "* https://realpython.com/python-web-scraping-practical-introduction/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "from selenium import webdriver\n",
    "import time\n",
    "from bs4 import BeautifulSoup\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Input Google login info to get into Medium\n",
    "USER = ''\n",
    "PASS = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logging in with Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Mediums statistics page is behind login, we'll have to automate this process in order to scrape the overview stats. This easiest way to do this in Python is widely viewed as Selenium. Selenium (not the chemical element) is a Python package built specifically for automating browsers. With a little bit of getting up to speed and learning to read through the html within the inspect view of Google chrome, you'll be up and running with automating anything your heart desires. For me, I struggled getting through Mediums email sign in due to a captcha, so I decided to use my google login and go that route. \n",
    "\n",
    "This will only work if you use your Google login to get into Medium, though it should be fairly easy to follow similar steps for Facebook, Twitter, etc. I import my username and password from another confidential python script. Feel free to do the same or just enter your information manually. Some code and resources that I drew from throughout this portion:\n",
    "* https://crossbrowsertesting.com/blog/test-automation/automate-login-with-selenium/\n",
    "* https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Goes through splash and login process\n",
    "def splash_process(driver, email, password):\n",
    "    \n",
    "    # Goes to sign in page\n",
    "    driver.get('https://medium.com/m/signin')\n",
    "\n",
    "    # Clicks sign in button\n",
    "    driver.find_element_by_xpath(\".//button[contains(.,'Sign in')]\").click()\n",
    "\n",
    "    # Clicks sign in with Google\n",
    "    driver.find_element_by_xpath(\".//button[contains(.,'Sign in with Google')]\").click()\n",
    "\n",
    "    # Finds email field\n",
    "    email_field = driver.find_element_by_id(\"identifierId\")\n",
    "\n",
    "    # Types in email\n",
    "    email_field.send_keys(email)\n",
    "\n",
    "    # Clicks next button\n",
    "    driver.find_element_by_id(\"identifierNext\").click()\n",
    "\n",
    "    # Wait a sec\n",
    "    time.sleep(1)\n",
    "\n",
    "    # Finds password field\n",
    "    pass_field = driver.find_element_by_name(\"password\")\n",
    "\n",
    "    # Types in password\n",
    "    pass_field.send_keys(password)\n",
    "\n",
    "    # Click next button\n",
    "    driver.find_element_by_id(\"passwordNext\").click()\n",
    "    \n",
    "    # Wait a sec\n",
    "    time.sleep(3)\n",
    "    \n",
    "    # Go to stats page and return it \n",
    "    driver.get('https://medium.com/me/stats')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Scrolls to bottom to get all posts into view\n",
    "def scroll(driver):\n",
    "\n",
    "    # Get scroll height\n",
    "    last_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
    "\n",
    "    while True:\n",
    "        # Scroll down to bottom\n",
    "        driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "\n",
    "        # Wait to load page\n",
    "        time.sleep(5)\n",
    "\n",
    "        # Calculate new scroll height and compare with last scroll height\n",
    "        new_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
    "        if new_height == last_height:\n",
    "            break\n",
    "        last_height = new_height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Start the driver\n",
    "driver = webdriver.Chrome('../chromedriver')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log in!\n",
    "splash_process(driver, USER, PASS)\n",
    "scroll(driver)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrangling HTML with BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And just like that we are in! Now we have to begin scraping the data from the raw html. There's a couple notable ways to do this, but the most common way in Python is usually to leverage the `BeautifulSoup` package. This allows us to take some raw html and parse it for specific things that we want. In my case, I'm looking for title, read time, publication, views, reads, ratio, and fans. Resources and threads of note for this section: \n",
    "* https://www.digitalocean.com/community/tutorials/how-to-scrape-web-pages-with-beautiful-soup-and-python-3\n",
    "* https://stackoverflow.com/questions/14444732/how-to-split-a-html-page-to-multiple-pages-using-python-and-beautiful-soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Grab the main table html from Medium stats \n",
    "table = driver.find_element_by_class_name('js-statsTableBody')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the raw html from our table element\n",
    "raw_html = table.get_attribute('innerHTML')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tr class=\"sortableTable-row sortableTable-row--dateBucket\"><td><span class=\"sortableTable-value\">1530208043198</span><header class=\"heading heading--allCaps\"><div class=\"u-clearfix\"><div class=\"heading-content u-floatLeft\"><span class=\"heading-title heading-title--bold\">June 2018</span></div></div></header></td></tr><tr class=\"sortableTable-row js-statsTableRow\" data-action=\"show-post-graph\" data-action-value=\"78d11456019\" data-timestamp=\"1530024573261\"><td class=\"sortableTable-rowTitle\"><span class=\"sortableTable-value\">1530208043198</span><h2 class=\"sortableTable-title\">An Ode to the Type A Data Scientist</h2><span class=\"sortableTable-text\"><span class=\"u-xs-hide\"><span class=\"readingTime\" title=\"7 min read\"></span><span class=\"middotDivider\"></span><em>In</em> <a class=\"sortableTable-link\" href=\"https://towardsdatascience.com\">Towards Data Science</a><span class=\"middotDivider\"></span></span><a class=\"sortableTable-link\" href=\"https://towardsdatascience.com/ode-to-the-type-a-data-scientist-78d11456019\">View story</a><span class=\"middotDivider\"></span><a class=\"sortableTable-link\" href=\"https://medium.com/p/78d11456019/referrers\">Referrers</a></span></td><td><span class=\"sortableTable-value\">3224</span><span class=\"sortableTable-number\" title=\"3,224\">3.2K<span class=\"u-sm-show\"><br>views</span></span></td><td><span class=\"sortableTable-value\">699</span><span class=\"sortableTable-number\" title=\"699\">699<span class=\"u-sm-show\"><br>reads</span></span></td><td><span class=\"sortableTable-value\">21.681141439205955</span><span class=\"sortableTable-number\">22%<span class=\"u-sm-show\"><br>ratio</span></span></td><td><span class=\"sortableTable-value\">72</span><span class=\"sortableTable-number\" title=\"72\">72<span class=\"u-sm-show\"><br>fans</span></span></td></tr><tr class=\"sortableTable-row js-statsTableRow\" data-action=\"show-post-graph\" data-action-value=\"df38552fc782\" data-timestamp=\"1530021639654\"><td class=\"sortableTable-rowTitle\"><span class=\"sortableTable-value\">15300\n"
     ]
    }
   ],
   "source": [
    "# Preview html\n",
    "print(raw_html[0:2000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Quit our driver\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clean html  \n",
    "soup = BeautifulSoup(raw_html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An Ode to the Type A Data Scientist\n"
     ]
    }
   ],
   "source": [
    "# Story titles\n",
    "titles = [item.text for i, item in enumerate(soup.select('h2'))]\n",
    "print(titles[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 min read\n"
     ]
    }
   ],
   "source": [
    "# Reading times\n",
    "read_times = [item.get('title') for i, item in enumerate(soup.findAll('span', {'class':'readingTime'}))]\n",
    "print(read_times[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Towards Data Science\n"
     ]
    }
   ],
   "source": [
    "# Publication names\n",
    "pubs = []\n",
    "h2tags = soup.find_all('h2')\n",
    "for h2tag in h2tags:\n",
    "    page = [str(h2tag)]\n",
    "    elem = h2tag.next_sibling\n",
    "    while elem and elem.name != 'h2':\n",
    "        if elem.text.split('View story')[0] == '':\n",
    "            pubs.append('None')\n",
    "        else:\n",
    "            pubs.append(elem.text.split('View story')[0][3::])\n",
    "        elem = elem.next_sibling\n",
    "print(pubs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get all numerical metrics\n",
    "nums = [item.text for i, item in enumerate(soup.findAll('span', {'class':'sortableTable-value'})) if (len(item.text) < 13 or '.' in item.text)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3224\n"
     ]
    }
   ],
   "source": [
    "# Views\n",
    "views = nums[::4]\n",
    "print(views[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "699\n"
     ]
    }
   ],
   "source": [
    "# Reads\n",
    "reads = nums[1::4]\n",
    "print(reads[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21.681141439205955\n"
     ]
    }
   ],
   "source": [
    "# Read ratio\n",
    "ratio = nums[2::4]\n",
    "print(ratio[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72\n"
     ]
    }
   ],
   "source": [
    "# Fans\n",
    "fans = nums[3::4]\n",
    "print(fans[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Pandas DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have lists of each particular feature that we wanted, it gets infinitely easier from here! We just have to adapt these lists into a DataFrame and then we can easily manipulate, utilize, and analyze our collected information for whatever purpose we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create dataframe\n",
    "df = pd.DataFrame(data={'Title': titles, 'Read Time': read_times, 'Publication': pubs, 'Views': views, \n",
    "                        'Reads': reads, 'Read Ratio': ratio, 'Fans': fans})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Reorder columns\n",
    "df = df[['Title', 'Publication', 'Read Time', 'Views', 'Reads', 'Read Ratio', 'Fans']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Publication</th>\n",
       "      <th>Read Time</th>\n",
       "      <th>Views</th>\n",
       "      <th>Reads</th>\n",
       "      <th>Read Ratio</th>\n",
       "      <th>Fans</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>An Ode to the Type A Data Scientist</td>\n",
       "      <td>Towards Data Science</td>\n",
       "      <td>7 min read</td>\n",
       "      <td>3224</td>\n",
       "      <td>699</td>\n",
       "      <td>21.681141439205955</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Self Driven Data Science - Issue #52</td>\n",
       "      <td>Hacker Noon</td>\n",
       "      <td>3 min read</td>\n",
       "      <td>283</td>\n",
       "      <td>115</td>\n",
       "      <td>40.63604240282685</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Choosing Your First Job: Size Matters</td>\n",
       "      <td>Hacker Noon</td>\n",
       "      <td>7 min read</td>\n",
       "      <td>391</td>\n",
       "      <td>144</td>\n",
       "      <td>36.828644501278774</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Python for Data Science and Machine Learning B...</td>\n",
       "      <td>Towards Data Science</td>\n",
       "      <td>6 min read</td>\n",
       "      <td>2175</td>\n",
       "      <td>795</td>\n",
       "      <td>36.55172413793103</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The Key to Optimal Internships</td>\n",
       "      <td>Hacker Noon</td>\n",
       "      <td>5 min read</td>\n",
       "      <td>197</td>\n",
       "      <td>80</td>\n",
       "      <td>40.609137055837564</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Title           Publication  \\\n",
       "0                An Ode to the Type A Data Scientist  Towards Data Science   \n",
       "1               Self Driven Data Science - Issue #52           Hacker Noon   \n",
       "2              Choosing Your First Job: Size Matters           Hacker Noon   \n",
       "3  Python for Data Science and Machine Learning B...  Towards Data Science   \n",
       "4                     The Key to Optimal Internships           Hacker Noon   \n",
       "\n",
       "    Read Time Views Reads          Read Ratio Fans  \n",
       "0  7 min read  3224   699  21.681141439205955   72  \n",
       "1  3 min read   283   115   40.63604240282685    7  \n",
       "2  7 min read   391   144  36.828644501278774   19  \n",
       "3  6 min read  2175   795   36.55172413793103   48  \n",
       "4  5 min read   197    80  40.609137055837564    9  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 76 entries, 0 to 75\n",
      "Data columns (total 7 columns):\n",
      "Title          76 non-null object\n",
      "Publication    76 non-null object\n",
      "Read Time      76 non-null object\n",
      "Views          76 non-null object\n",
      "Reads          76 non-null object\n",
      "Read Ratio     76 non-null object\n",
      "Fans           76 non-null object\n",
      "dtypes: object(7)\n",
      "memory usage: 4.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# Overview\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our information neatly placed in a Pandas DataFrame, we can't ignore data cleaning. Not every dataset will be squeaky clean like Kaggle may lead you to believe. This is especially applicable when dealing with scraped data where anomalies can come up from time to time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 76 entries, 0 to 75\n",
      "Data columns (total 7 columns):\n",
      "Title          76 non-null object\n",
      "Publication    76 non-null object\n",
      "Read Time      76 non-null int64\n",
      "Views          76 non-null int64\n",
      "Reads          76 non-null int64\n",
      "Read Ratio     76 non-null float64\n",
      "Fans           76 non-null int64\n",
      "dtypes: float64(1), int64(4), object(2)\n",
      "memory usage: 4.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# Convert numerical features to floats\n",
    "df = df.apply(pd.to_numeric, errors='ignore')\n",
    "df['Read Time'] = df['Read Time'].apply(lambda x: int(x.split()[0]))\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Publication</th>\n",
       "      <th>Read Time</th>\n",
       "      <th>Views</th>\n",
       "      <th>Reads</th>\n",
       "      <th>Read Ratio</th>\n",
       "      <th>Fans</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>The 5 Proven Best Ways to Start Your Day</td>\n",
       "      <td>The Ascent</td>\n",
       "      <td>4</td>\n",
       "      <td>109</td>\n",
       "      <td>69</td>\n",
       "      <td>63.302752</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>5 Lessons From Going Idea to Prototype in 24 H...</td>\n",
       "      <td>UX Planet</td>\n",
       "      <td>5</td>\n",
       "      <td>1454</td>\n",
       "      <td>616</td>\n",
       "      <td>42.365887</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Self Driven Data Science — Issue #5</td>\n",
       "      <td>Towards Data Science</td>\n",
       "      <td>2</td>\n",
       "      <td>253</td>\n",
       "      <td>146</td>\n",
       "      <td>57.707510</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>A Beginners Guide to the World Within Data Sci...</td>\n",
       "      <td>Towards Data Science</td>\n",
       "      <td>13</td>\n",
       "      <td>1478</td>\n",
       "      <td>334</td>\n",
       "      <td>22.598106</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>How I Created My Personal Logo as an Amateur D...</td>\n",
       "      <td>UX Planet</td>\n",
       "      <td>5</td>\n",
       "      <td>601</td>\n",
       "      <td>384</td>\n",
       "      <td>63.893511</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Title           Publication  \\\n",
       "71           The 5 Proven Best Ways to Start Your Day            The Ascent   \n",
       "72  5 Lessons From Going Idea to Prototype in 24 H...             UX Planet   \n",
       "73                Self Driven Data Science — Issue #5  Towards Data Science   \n",
       "74  A Beginners Guide to the World Within Data Sci...  Towards Data Science   \n",
       "75  How I Created My Personal Logo as an Amateur D...             UX Planet   \n",
       "\n",
       "    Read Time  Views  Reads  Read Ratio  Fans  \n",
       "71          4    109     69   63.302752    10  \n",
       "72          5   1454    616   42.365887    28  \n",
       "73          2    253    146   57.707510     4  \n",
       "74         13   1478    334   22.598106    29  \n",
       "75          5    601    384   63.893511     9  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Another look\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Read Time</th>\n",
       "      <th>Views</th>\n",
       "      <th>Reads</th>\n",
       "      <th>Read Ratio</th>\n",
       "      <th>Fans</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>76.000000</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>76.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.052632</td>\n",
       "      <td>2975.539474</td>\n",
       "      <td>763.947368</td>\n",
       "      <td>46.335564</td>\n",
       "      <td>60.644737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.952057</td>\n",
       "      <td>11670.325013</td>\n",
       "      <td>2304.829156</td>\n",
       "      <td>12.793347</td>\n",
       "      <td>226.094476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>109.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>10.426731</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>344.500000</td>\n",
       "      <td>146.750000</td>\n",
       "      <td>40.432070</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>603.000000</td>\n",
       "      <td>282.000000</td>\n",
       "      <td>49.007654</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.000000</td>\n",
       "      <td>1096.500000</td>\n",
       "      <td>441.000000</td>\n",
       "      <td>56.062458</td>\n",
       "      <td>27.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>13.000000</td>\n",
       "      <td>85373.000000</td>\n",
       "      <td>14945.000000</td>\n",
       "      <td>64.351852</td>\n",
       "      <td>1447.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Read Time         Views         Reads  Read Ratio         Fans\n",
       "count  76.000000     76.000000     76.000000   76.000000    76.000000\n",
       "mean    4.052632   2975.539474    763.947368   46.335564    60.644737\n",
       "std     1.952057  11670.325013   2304.829156   12.793347   226.094476\n",
       "min     2.000000    109.000000     60.000000   10.426731     1.000000\n",
       "25%     3.000000    344.500000    146.750000   40.432070     9.000000\n",
       "50%     3.000000    603.000000    282.000000   49.007654    14.000000\n",
       "75%     5.000000   1096.500000    441.000000   56.062458    27.250000\n",
       "max    13.000000  85373.000000  14945.000000   64.351852  1447.000000"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Numerical overview\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our clean dataset, let's export it as a csv file that we'll use for exploratory data analysis in the next portion of this project where we'll dive into the more subtle aspects of analyzing engagement in respect to my work and later on a larger scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Export as csv\n",
    "df.to_csv('mystats.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping Up\n",
    "That's all for the scraping process. Check out the follow up notebook titled `Medium Stats Data Analysis` in this repo as well. There is also a script that you can run and largely automate this process if you're interested in skipping straight the the analysis. Thanks for reading! Follow me on Medium if interested!\n",
    "\n",
    "https://medium.com/@conordewey3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}