LIWC-report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Developing the Western Business Sentiment Dictionary - Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to power an artificial intelligence program able to analyze vast amounts of text and assess the related sentiment, we needed to have human subjects report their reaction to business-related textual excerpts. We had 190 participants who signed up for one of the available time slots to complete the survey at Ivey's Behavioural Lab. Participants went to the Lab at the scheduled time and for 60 minutes (with an 8-minute break in between) they assessed as many sentences as they could using the PyBossa crowdsourcing platform.\n",
    "\n",
    "The main goal of this project was to create a reliable sentiment classifier for the business domain. Our working hypothesis was that due to the complexity and rhetoric of the business world, current industry solutions for sentiment analysis would perform poorly. We used the renowned LIWC as the de facto standard to evaluate these texts and compare its results with our pool of participants.\n",
    "\n",
    "This work was performed by the [CulturePlex Lab](http://www.cultureplex.ca/) with the contributions of Antonio Jiménez-Mavillard, Javier de la Rosa, Adriana Soto-Corominas, Juan Luis Suárez. The code of this work can be found [here](http://nbviewer.ipython.org/github/mavillard/liwc/blob/master/comparison.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##2. LIWC's results vs participants' results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###a) Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset consisted of 8,996 sentences obtained by searching specific terms (see [Appendix A](#Appendix-A)) from The New York Times API and retrieving the related sentences from articles published in 2013 that contained those terms.\n",
    "\n",
    "LIWC contains a dictionary of 4,500 words classified in various categories. One word can belong to several categories. For example, the word *cried* is part of five word categories: sadness, negative emotion, overall affect, verb, and past tense verb. Many of the LIWC categories are arranged hierarchically. For instance, all anger words, by definition, are categorized as negative emotion and overall emotion words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###b) Sentence scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Participants' assessment\n",
    "The sentences were evaluated by 190 participants with values from the range [-2 (very negative), -1 (negative), 0 (neutral), +1 (positive), +2 (very positive)]. Not every participant evaluated all the senteces but every sentence was evaluated at least three times (by three different participants).\n",
    "\n",
    "The final score per sentence was the average of its individual evaluations. Therefore, every sentence got a final score between -2 and +2. Finally, this score was normalized to the range [-1, +1]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####LIWC scores\n",
    "\n",
    "LIWC analyzes every sentence word by word. As each target word is processed, the dictionary is searched, looking for a dictionary match with the current target word. If the target word matches the category word, the appropriate word category scale (or scales) for that word is incremented. Due to its hierarchical category arrangement, only the two top categories for emotions (positive emotion and negative emotion) were considered for this experiment.\n",
    "\n",
    "In order to estimate the soundness of LIWC at analyzing the sentiment of our dataset, this software was set up with the following options:\n",
    "- Metrics:\n",
    "    - posemo: % of words that express a positive emotion\n",
    "    - negemo: % of words that express a negative emotion\n",
    "- Segment delimiter:\n",
    "    - newline: original file splitted into individual lines, i. e., the analysis was performed sentence by sentence\n",
    "- Tokens:\n",
    "    - words: the words were the only type of token taken into account; numerals and punctuation marks were ignored\n",
    "\n",
    "Upon completing the analysis, every sentence got a value for posemo, between 0 and 100 (%), and another value for negemo, also between 0 and 100 (%). This two metrics were used to calculate the final score per sentence by considering two options:\n",
    "- Option 1) Subtraction:\n",
    "    - score = posemo - negemo\n",
    "- Option 2) Maximum:\n",
    "    - if posemo > negemo, then score = posemo\n",
    "    - if posemo < negemo, then score = -negemo\n",
    "    - if posemo = negemo, then score = 0\n",
    "\n",
    "In both cases, the final score for every sentence was between -100 and +100. They both are the extreme cases in which all the words in the sentence are negative (-100) or all the words are positive (+100). In order to normalize, two different criteria were followed. One option was to normalize from this full range, [-100, +100], to [-1, +1]. But actually, the most positive sentence evaluated by LIWC got a score of ~ +25 while the most negative one got a score of ~ -24, so a fairer way to normalize LIWC scores was from the maximum values observed, [-25, +25], to [-1, +1]. These two options are a) and b) respectively and are formally defined as follows:\n",
    "\n",
    "- Option a) Normalizing from the full range:\n",
    "    - [-100, +100] -> [-1, +1]\n",
    "- Option b) Normalizing from the maximum range observed. Let mpe be the maximum value observed for posemo, mne the maximum value observed for negemo, among all the sentences, and m the maximum value between mpe and mne; then:\n",
    "    - [-m, +m] -> [-1, +1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###c) Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "- norm_liwc = LIWC normalized score\n",
    "- norm_score = Participants normalized score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Histograms\n",
    "The next images show the participants' frequency distributions for the 8996 sentences compared to each of the four different combinations (options a) b) 1) and 2)) of LIWC's distributions. The normalized range ([-1, +1]) was splitted into 15 bins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_a1_sep.png\"/>\n",
    "        <figcaption>Fig.1 Participants (rigth) vs LIWC (left) with options a) full range 1) subtraction</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_a1_tog.png\"/>\n",
    "        <figcaption>Fig.2 Participants vs LIWC (overlaid) with options a) full range 1) subtraction</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_a2_sep.png\"/>\n",
    "        <figcaption>Fig.3 Participants (rigth) vs LIWC (left) with options a) full range 2) maximum</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_a2_tog.png\"/>\n",
    "        <figcaption>Fig.4 Participants vs LIWC (overlaid) with options a) full range 2) maximum</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b1_sep.png\"/>\n",
    "        <figcaption>Fig.5 Participants (rigth) vs LIWC (left) with options b) maximum range observed 1) subtraction</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b1_tog.png\"/>\n",
    "        <figcaption>Fig.6 Participants vs LIWC (overlaid) with options b) maximum range observed 1) subtraction</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b2_sep.png\"/>\n",
    "        <figcaption>Fig.7 Participants (rigth) vs LIWC (left) with options b) maximum range observed 2) maximum</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b2_tog.png\"/>\n",
    "        <figcaption>Fig.8 Participants vs LIWC (overlaid) with options b) maximum range observed 2) maximum</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous histograms note how different are LIWC results from the participant results. LIWC is strongly biased towards 0 (it evaluates a large amount of sentences as neutral). The second remarkable point is that LIWC rarely classifies the sentence as very negative or very possitive, in contrast to the participants.\n",
    "\n",
    "The rest of the analysis was done considering only the option b) 2), the most favorable to LIWC as it is the one that provides the best accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Normality tests\n",
    "Despite the fact that the images above seem to show normal distributions, normality tests reveal the opposite. The next section shows some normality performed to both distributions.\n",
    "\n",
    "<table>\n",
    "    <caption>Tab.1 Normality tests</caption>\n",
    "    <thead>\n",
    "        <th>name</th><th>distribution</th><th>p-value</th>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr><td>Anderson-Darling</td><td>participants</td><td>0.0</td></tr>\n",
    "        <tr><td>Anderson-Darling</td><td>LIWC</td><td>0.0</td></tr>\n",
    "        <tr><td>Kolmogorov-Smirnov</td><td>participants</td><td>4.63e-121</td></tr>\n",
    "        <tr><td>Kolmogorov-Smirnov</td><td>LIWC</td><td>0.0</td></tr>\n",
    "        <tr><td>Pearson</td><td>participants</td><td>1.39e-26</td></tr>\n",
    "        <tr><td>Pearson</td><td>LIWC</td><td>4.83e-181</td></tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "\n",
    "For every test, p-value < 0.05, i. e., neither of the distributions are normal. This is not a surprising result as \"data from the human social world is often not normally distributed.\" (John Canning, \"[Statistics for the Humanities](http://statisticsforhumanities.net/book/)\", 2014)\n",
    "\n",
    "In order to quantify how different both distributions are, more analysis (KDE, QQ plot, and Kolmogorov-Smirnov for two samples) were done. The results are shown in the next subsections:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Kernel density estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b2_den.png\"/>\n",
    "        <figcaption>Fig.9 KDE: participants' vs LIWC's kernel density estimation</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure 9 shows a result very similar to the histograms: LIWC is strongly biased towards 0, while the natural is a bias towards positive values (\"[Human language is biased towards happiness, say computational linguists](https://medium.com/the-physics-arxiv-blog/data-mining-reveals-how-human-language-is-biased-towards-happiness-773df682c4a7)\", 2014), and besides, it rarely classifies the sentence as very negative or very possitive, in contrast to the students."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Quantile-quantile plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"qqplot-students.png\"/>\n",
    "        <figcaption>Fig.10 QQPlot: participants' distribution (blue dots) against the normal distribution (red line)</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"qqplot-liwc.png\"/>\n",
    "        <figcaption>Fig.11 QQPlot: LIWC's distribution (blue dots) against the normal distribution (red line)</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figures 10 and 11 show how different both distributions are: the participants' distribution is very close to the normal distribution while LIWC's is very far."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####Kolmogorov-Smirnov (two samples)\n",
    "The comparison of the two distributions performed with the test provided by scipy returned a value for p-value=0.0, i. e., participants' and LIWC's results come from different distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "###d) Accuracy\n",
    "The accuracy of LIWC's result with respect to the polarity assigned by the participants is:\n",
    "\n",
    "<div>\n",
    "    <table>\n",
    "        <tr><td>Total sentences</td><td>8996</td></tr>\n",
    "        <tr><td>Sentences evaluated with same polarity</td><td>3202</td></tr>\n",
    "        <tr><td>Sentences evaluated with different polarity</td><td>5794</td></tr>\n",
    "        <tr><td>LIWC accuracy</td><td>35.59%</td></tr>\n",
    "        <caption>LIWC accuracy</caption>\n",
    "    </table>\n",
    "</div>\n",
    "\n",
    "LIWC accuracy (35.59%) is very below the minimum value admissible for any classificator (70%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "###e) Conclusions\n",
    "A variety of tests were performed in order to compare how similar or different were the polarities assigned to the sentences by the participants and by LIWC. Both distributions are slightly biased towards a positive polarity (as the histograms and the density estimations show); however, there are much more important differences:\n",
    "1. The histograms shows that LIWC is strongly biased towards 0 (it evaluates a large amount of sentences as neutral) and it rarely classifies the sentence as very negative or very possitive, in contrast to the participants.\n",
    "2. The KDE reinforces the results obtained from the histograms.\n",
    "3. The QQPlot shows how different both distributions are with respect to the normal distribution: the participants' distribution is very close while LIWC's is very far from the normal distribution.\n",
    "4. Kolmogorov-Smirnov gives a definitive result: participants' and LIWC's results come from different distributions.\n",
    "5. LIWC accuracy (35.59%) is very below the minimum value admissible for any classificator (70%).\n",
    "\n",
    "Therefore, the conclusion is that it is necessary to create a more suitable classifier for this case study."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##3. Demographic data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next images shows information on demographics. As each participant evaluated a different subset of sentences, it is not possible to establish a direct correlation between demographic data (participants' country, gender, or age) and sentences' polarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 11 shows the distribution of participants by country."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"total_users.png\"/>\n",
    "        <figcaption>Fig.11 Distribution of participants by country</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 12 shows the global participants' evaluations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"total_evals.png\"/>\n",
    "        <figcaption>Fig.12 Global participants' sentence evaluation</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 13 shows the participants' evaluations by country and gender."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"demo.png\"/>\n",
    "        <figcaption>Fig.13 Participants' sentence evaluation by country and gender</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Appendix A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "entrepreneur\n",
    "startup\n",
    "new venture\n",
    "manager\n",
    "executive\n",
    "founder\n",
    "wal-mart\n",
    "exxon\n",
    "chevron\n",
    "berkshire hathaway\n",
    "apple\n",
    "samsung\n",
    "phillips 66\n",
    "general motors\n",
    "ford\n",
    "valero\n",
    "exelon corporation\n",
    "dow chemical\n",
    "conocophillips\n",
    "morgan chase\n",
    "world fuel services\n",
    "illinois tool works\n",
    "boeing\n",
    "intel corporation\n",
    "pepsico\n",
    "tjx companies\n",
    "costco wholesale\n",
    "metlife\n",
    "blackrock\n",
    "at&t\n",
    "johnson & johnson\n",
    "abbott laboratories\n",
    "cvs caremark\n",
    "visa\n",
    "cisco systems\n",
    "walt disney company\n",
    "rock tenn company\n",
    "halliburton company\n",
    "alcoa\n",
    "fluor corporation\n",
    "general electric\n",
    "cbre group\n",
    "energy transfer equity\n",
    "express scripts holding\n",
    "hca holdings\n",
    "nrg energy\n",
    "aramark holdings corporation\n",
    "honeywell international\n",
    "nike\n",
    "amazon.com\n",
    "procter & gamble\n",
    "unitedhealth group\n",
    "microsoft\n",
    "Google\n",
    "Salesforce\n",
    "SAS\n",
    "Quicken\n",
    "Intuit\n",
    "las vegas sands\n",
    "autonation\n",
    "3m company\n",
    "danaher corporation\n",
    "new york life insurance company\n",
    "mcdonalds corporation\n",
    "caterpillar\n",
    "ryder systems\n",
    "d.r. horton\n",
    "stanley black & decker\n",
    "archer daniels midland\n",
    "coca-cola company\n",
    "united continental holdings\n",
    "ingram micro\n",
    "chs\n",
    "international paper company\n",
    "mckesson\n",
    "phillip morris international\n",
    "state farm insurance\n",
    "owens corning\n",
    "waste management\n",
    "c.h. robinson worldwide\n",
    "harley-davidson\n",
    "manpower group\n",
    "union pacific\n",
    "r.r donnelley & sons company\n",
    "graham holdings company\n",
    "emc corporation\n",
    "mattel\n",
    "united parcel service\n",
    "omnicom group\n",
    "mckinsey & company\n",
    "pricewaterhousecoopers\n",
    "intuitive research and tech\n",
    "international bussiness machines\n",
    "formerly BrightTag\n",
    "aka Wright Foods\n",
    "turospree.com\n",
    "nirvanix.com\n",
    "getpostrocket.com\n",
    "votertide.com\n",
    "gamelayers.com\n",
    "unifyo.com\n",
    "lookery.com\n",
    "canv.as\n",
    "blurtt.com\n",
    "manilla.com\n",
    "admaze.ly.com\n",
    "springpad.com\n",
    "ex.fm\n",
    "sambamobile.com\n",
    "qik.com\n",
    "inbloom.org\n",
    "intonow.com\n",
    "getfindit.com\n",
    "inqmobile.com\n",
    "outboxmail.com\n",
    "argylesocial.com\n",
    "iamexec.com\n",
    "stippleit.com\n",
    "500px Inc\n",
    "Wajam\n",
    "Keek Inc\n",
    "GuySpy\n",
    "Woozworld\n",
    "TinEye\n",
    "Idee Inc\n",
    "Jigsy\n",
    "Voices.com\n",
    "Snapsort\n",
    "EmpireAvenue\n",
    "Frank and Oak\n",
    "LoginRadius\n",
    "Chango Inc\n",
    "Pure Energies Group Inc\n",
    "Buytopia.ca\n",
    "Smoke NV Inc\n",
    "Just Quality International Inc\n",
    "Sterling Cross Defense Systems Corp\n",
    "Magnet Forensics Inc\n",
    "Greenlife Water Corp\n",
    "Secure Sense Solutions\n",
    "Sandstorm Gold Ltd\n",
    "ScribbleLive Technologies Inc\n",
    "TDot Performance Ltd\n",
    "Redwood Strategic Inc\n",
    "Emcision International Inc\n",
    "Henkaa Inc\n",
    "Whirlscape Inc\n",
    "Bionym\n",
    "Thalmic Labs\n",
    "Recon Instruments\n",
    "Om Malik Technology\n",
    "Affino\n",
    "AnalyzeRE\n",
    "Atomic Reach\n",
    "Cloud Dynamics\n",
    "Exo U\n",
    "Fabric Engine\n",
    "Finmaven\n",
    "Granify\n",
    "InfraDog\n",
    "Introhive\n",
    "Invici Technologies\n",
    "Login Radius\n",
    "mnubo\n",
    "Obero Solutions\n",
    "PeopleInsight\n",
    "Procima Experts\n",
    "reelyActive\n",
    "RtTech Software\n",
    "Seamless Mobile Health\n",
    "Slyce\n",
    "Smart Employee Benefits\n",
    "SweetIQ\n",
    "Thalmic Labs\n",
    "Vena Solutions\n",
    "WestonExprerssions\n",
    "Linkett\n",
    "360pi\n",
    "AcademixDirect Inc\n",
    "Adconion Direct\n",
    "Adesto Technologies Corporation\n",
    "Alfresco\n",
    "Aitierre Corporation\n",
    "Altitude Digital\n",
    "Anametrix\n",
    "Aphios Corporation\n",
    "AppDirect\n",
    "Applied Predictive Technologies\n",
    "appsFreedom\n",
    "Apttus\n",
    "Argos Therapeutics Inc\n",
    "Aseptia/Wright Foods\n",
    "Atlantis Computing\n",
    "Avere Systems\n",
    "Become Inc\n",
    "BeyondTrust\n",
    "Bluebeam Software Inc\n",
    "Signal\n",
    "Cinsay Inc\n",
    "Cognitive Networks\n",
    "Compass-EOS\n",
    "Credit Sesame\n",
    "Curemark LLC\n",
    "Daintree Networks\n",
    "Data Dynamics Inc\n",
    "Defense.Net\n",
    "DigiCert Inc\n",
    "DiscGenics\n",
    "Duda\n",
    "Electric Cloud\n",
    "Flybits\n",
    "Fruition Partners\n",
    "GENBAND\n",
    "globalVCard\n",
    "CSI Enterprise\n",
    "GuideSpark\n",
    "HasOffers\n",
    "Health Catalyst\n",
    "Hortonworks\n",
    "iBuildApp\n",
    "iCIMS Inc\n",
    "ImaginAb Inc\n",
    "iPipeline\n",
    "JAB Broadband\n",
    "Kahuna Inc\n",
    "Kareo\n",
    "Klocwork Inc.\n",
    "Rogue Wave\n",
    "Lancope\n",
    "Load DynamiX\n",
    "LocalResponse\n",
    "Looker\n",
    "Malcovery Security\n",
    "Maxta Inc\n",
    "MicroGREEN Polymers Inc\n",
    "Modernizing Medicine Inc\n",
    "myoscience\n",
    "NearWoo\n",
    "Nomis Solutions Inc\n",
    "OxiCool Inc\n",
    "Packsize LLC\n",
    "Panaya\n",
    "Paxata\n",
    "Perfecto Mobile\n",
    "Perseus Telecom\n",
    "Phunware\n",
    "Pluribus Networks\n",
    "PowerPlan Inc\n",
    "Quanterix\n",
    "RealMatch\n",
    "Red Lambda Inc\n",
    "Rise Interactive\n",
    "RootMetrics\n",
    "RxWiki Inc\n",
    "Scaled Agile Inc\n",
    "Scribe Software\n",
    "Seamless Medical Systems Inc\n",
    "SecureAuth\n",
    "SeeChange Health\n",
    "Shape Security\n",
    "SkyCross\n",
    "Skyhigh Networks\n",
    "Skyonic Corporation\n",
    "SmartZip Analytics Inc\n",
    "Solace Systems\n",
    "Solavei\n",
    "SpiderCloud Wireless Inc\n",
    "Splice Machine\n",
    "SpotXchange Inc\n",
    "SPR Therapeutics LLC\n",
    "Stoke Inc\n",
    "StreetLight Data\n",
    "Sureline Systems Inc\n",
    "T3 Motion Inc\n",
    "TechLeads Online\n",
    "Tenable Network Security Inc\n",
    "ThreatTrack Security\n",
    "Trueffect\n",
    "TRX Systems Inc\n",
    "Ubiquity Global Services Inc\n",
    "Welltok\n",
    "Xactly Corporation\n",
    "XCOR Aerospace Inc\n",
    "Yashi\n",
    "Zeta Interactive\n",
    "Fuh\n",
    "Quest Nutrition\n",
    "Reliant Asset Management\n",
    "Superfish\n",
    "Acacia Communications\n",
    "Provider Power\n",
    "Crescendo Bioscience\n",
    "Plexus Worldwide\n",
    "Vacasa\n",
    "Go Energies\n",
    "Minute Key\n",
    "Sainstore\n",
    "The HCI Group\n",
    "Dynamic Dental Partners Group\n",
    "Aseptia\n",
    "Asentra Health\n",
    "American Solar Direct\n",
    "Prescient Edge\n",
    "BlueSquare Resolutions\n",
    "BES Design Build\n",
    "Simpler Trading\n",
    "sweetFrog Premium Frozen Yogurt\n",
    "Base Commerce\n",
    "CPSG Partners\n",
    "MedHOK\n",
    "Showroom Logic\n",
    "WeVeel\n",
    "PackIt\n",
    "American Retirement Advisors\n",
    "Highpoint Global AdKarma\n",
    "Axtria\n",
    "Dolls Kill\n",
    "Centric Digital\n",
    "Retail Capital\n",
    "Restore Health\n",
    "OpenRoad Lending\n",
    "Pontchartrain Partners\n",
    "Team Extreme Marketing International\n",
    "Choice Energy\n",
    "ReviMedia\n",
    "SmartZip Analytics\n",
    "Thompson Gray\n",
    "MCSG Technologies\n",
    "Innovative Surveillance Solutions\n",
    "Nordic\n",
    "Michigan Realty Solutions\n",
    "Ohio Investments\n",
    "Adore Me\n",
    "IQ Formulations\n",
    "Alcohol by Volume\n",
    "H.Bloom\n",
    "Tealium\n",
    "Boostability\n",
    "Solve Media\n",
    "Heartland Energy Partners\n",
    "Bridger\n",
    "Jeunesse Global\n",
    "Touchsuite\n",
    "Accleration Partners\n",
    "Moore Heating & Air Conditioning\n",
    "Optimatic Media\n",
    "EPI Engineering\n",
    "Simplicity Laser\n",
    "Cinium Financial Services\n",
    "TNH Advanced Specialty Pharmacy\n",
    "Human Movement Management\n",
    "Allegheny Science & Technology\n",
    "Hudl\n",
    "GuideSpark\n",
    "Optimus Technology\n",
    "Luxe Royale\n",
    "SoluComp\n",
    "RPM Freight Systems\n",
    "Motivational Press\n",
    "99dresses\n",
    "Dinnr\n",
    "Seesmic Video\n",
    "Treehouse Logic\n",
    "Backchat\n",
    "Patient Communicator\n",
    "Twitpic\n",
    "BERG\n",
    "Wishberg\n",
    "GreenGar Studios\n",
    "Rivet & Sway\n",
    "Dijiwan\n",
    "Wantful\n",
    "Disruptive Media\n",
    "Calexda\n",
    "TurnTable.fm\n",
    "Tutorspree\n",
    "Nirvanix\n",
    "PostRocket\n",
    "VoterTide\n",
    "SkyRocket\n",
    "GameLayers\n",
    "Serendip\n",
    "Unifyo\n",
    "Lookery\n",
    "Canvas Networks\n",
    "Blurtt\n",
    "Manilla\n",
    "Pumodo\n",
    "Admazely\n",
    "Springpad\n",
    "ExFM\n",
    "Samba Mobile\n",
    "Qik\n",
    "inBloom\n",
    "IntoNow\n",
    "FindIt\n",
    "Inq Mobile\n",
    "Outbox\n",
    "Argyle Company\n",
    "Exec\n",
    "Bloom.fm\n",
    "Stipple\n",
    "Zumbox\n",
    "Behance\n",
    "Eventbrite\n",
    "MailChimp\n",
    "Twitch\n",
    "Moz\n",
    "HubSpot\n",
    "Indeed\n",
    "BufferApp\n",
    "Airbnb\n",
    "ThinkGeek\n",
    "Quora\n",
    "Tesla Motors\n",
    "Grooveshark\n",
    "Klout\n",
    "Zendesk\n",
    "About.me\n",
    "Uber\n",
    "Lumosity"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}