cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🧹Joblisting Cleaning 🧹"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "Table of Contents: <br>\n",
    "    \n",
    "<ul>\n",
    "    <li>1. <a href=\"#1.-Setup\">Setup</a></li>\n",
    "    <li>2. <a href=\"#2.-Loading-the-Data\">Loading the Data</a></li>\n",
    "    <li>3. <a href=\"#3.-Framing-the-Problem\">Framing the Problem</a>\n",
    "        <ul>\n",
    "            <li>3.1. <a href=\"#3.1.-Objective\">Objective</a></li>\n",
    "            <li>3.2. <a href=\"#3.2.-Procedure\">Procedure</a></li>\n",
    "            <li>3.3. <a href=\"#3.3.-Design\">Design</a></li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    <li>4. <a href=\"#4.-Previewing-the-Data\">Previewing the Data</a></li>\n",
    "    <li>5. <a href=\"#5.-Wrangling-and-Cleaning-the-Data\">Wrangling and Cleaning the Data</a>\n",
    "        <ul>\n",
    "            <li>5.1. <a href=\"#5.1.-company-and-rating\">company and rating</a></li>\n",
    "            <li>5.2. <a href=\"#5.2.-headquarters\">headquarters</a></li>\n",
    "            <li>5.3. <a href=\"#5.3-salary-estimate\">salary estimate</a></li>\n",
    "            <li>5.4. <a href=\"#5.4-job-type\">job type</a></li>\n",
    "            <li>5.5. <a href=\"#5.5.-size,-founded,-type,-industry,-sector,-revenue,-and-job-description\">size, founded, type, industry, sector, revenue, and job description</a></li>\n",
    "            <li>5.6. <a href=\"#5.6.-Building-the-Pipeline\">Building the Pipeline</a></li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    <li>6. <a href=\"#6.-Saving\">Saving</a></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ If the following assertions fail, I believe other versions still work. Just ensure you are not using too outdated libraries (e.g. Python 3.* compared to Python 2.*). These are just the versions I'm using for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<frozen importlib._bootstrap>:219: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject\n"
     ]
    }
   ],
   "source": [
    "# General imports.\n",
    "\n",
    "# Python ≥ 3.7.9 is used.\n",
    "import sys\n",
    "assert sys.version_info >= (3, 7, 9)\n",
    "\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "from collections import Counter\n",
    "from ordered_set import OrderedSet\n",
    "\n",
    "# Specific imports.\n",
    "\n",
    "# NumPy ≥ 1.19.5 is used.\n",
    "import numpy as np\n",
    "assert np.__version__ >= \"1.18.5\"\n",
    "\n",
    "# Pandas ≥ 1.2.4 is used.\n",
    "import pandas as pd\n",
    "assert pd.__version__ >= \"1.2.4\"\n",
    "\n",
    "# Scikit-learn ≥ 0.24.2 is used.\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.24.2\"\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.base import TransformerMixin, BaseEstimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility Function(s).\n",
    "\n",
    "# Borrowed from: \n",
    "# https://github.com/ageron/handson-ml2/blob/master/04_training_linear_models.ipynb.\n",
    "\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"img\")\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(ax, fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    ax.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Loading the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = \"../input/joblisting.csv\"\n",
    "df = pd.read_csv(PATH, index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Framing the Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Objective\n",
    "\n",
    "+ Wrangle, clean, and perform Exploratory Data Analysis (EDA) on the Glassdoor.com joblisting data. Extract useful findings in relationships between estimated salary and other variables. I'll split this EDA into 2 parts: descriptive and exploratory analysis for the first part and question-driven exploration for the second part.\n",
    "\n",
    "We will work on wrangling and cleaning in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Procedure\n",
    "\n",
    "1. Setup.\n",
    "2. Loading the data.\n",
    "3. Framing the problem.\n",
    "4. Preview the data.\n",
    "5. Wrangling and cleaning the data.\n",
    "6. Univariate non-graphical analysis.\n",
    "7. Univariate graphical analysis.\n",
    "8. Multivariate non-graphical analysis. \n",
    "9. Multivariate graphical analysis.\n",
    "10. Organize findings.\n",
    "\n",
    "Disclaimer: Steps aren't necessarily done sequentially (I may jump back and forth a bit if I come up with an idea about something)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Design\n",
    "\n",
    "+ 8 total Google sheets will be continually updated throughout the project: 6 sheets for questions, assumption_univariate, conclusion_univariate, and observation_univariate (assumption_multivariate, conclusion_multivariate, observation_multivariate). The last sheet is called miscellaneous_observations and will contain observations about things other than the features themselves. These are my **analysis log sheets**.\n",
    "\n",
    "|              | assumptions              | conclusions              | observations              |\n",
    "| :----------- | :----------------------- | :----------------------- | :------------------------ |\n",
    "| univariate   | assumptions_univariate   | conclusions_univariate   | observations_univariate   |\n",
    "| multivariate | assumptions_multivariate | conclusions_multivariate | observations_multivariate |\n",
    "\n",
    "| Other                      |\n",
    "| :------------------------- |\n",
    "| miscellaneous_observations |\n",
    "| Questions                  |\n",
    "\n",
    "+ Assumptions will be updated mostly at the beginning of the project (can still be updated all throughout the project).\n",
    "+ Conclusions will mostly be done during and at the end of the project (can still be updated all throughout the project). \n",
    "+ Observations will, unlike the previous 2, be updated all throughout the project.\n",
    "\n",
    "\n",
    "+ For this project, I created a diagram to go along and keep track of all the edits I've made. I call this my **edit log**.\n",
    "+ **Note**: for the 6 sheets I showed in the markdown table above, the assumptions, conclusions, and observations will be denoted by [row, column] and for miscellaneous_observations and Questions sheets, I will denote it by [row,]. For multivariate cases, the number `row` will simply denote the count.\n",
    "+ **Note**: Assumptions are kept exclusively on the spreadsheet, Questions are denoted with (❓), conclusions with (📑), and observations with (🔍).\n",
    "\n",
    "\n",
    "+ I'll try to adhere by Python conventions in PEP8 and [this](https://stackoverflow.com/questions/43577404/purpose-of-import-this) (though I can't promise this will turn out perfectly)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Previewing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>job title</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>founded</th>\n",
       "      <th>type</th>\n",
       "      <th>industry</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "      <th>job description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Walmart\\n3.4</td>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>1994</td>\n",
       "      <td>company - public</td>\n",
       "      <td>general merchandise &amp; superstores</td>\n",
       "      <td>retail</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "      <td>Position Summary...\\nWhat you'll do...\\nAnalyt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TikTok\\n3.8</td>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>501 to 1000 employees</td>\n",
       "      <td>2016</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "      <td>TikTok is the leading destination for short-fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>Principal Data Scientist - Candidate Recommend...</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$187K - $231K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>2004</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "      <td>Your Job\\nThe Candidate Recommendations team b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>Senior Data Scientist - Moderation Engineering</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$130K - $156K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>2004</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "      <td>Your Job\\nThe Moderation Engineering team’s mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America\\n3.8</td>\n",
       "      <td>Data Scientist III</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>1902</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "      <td>Thermo Fisher Scientific Inc. is the world lea...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        company  \\\n",
       "0                  Walmart\\n3.4   \n",
       "1                   TikTok\\n3.8   \n",
       "2                   Indeed\\n4.3   \n",
       "3                   Indeed\\n4.3   \n",
       "4  Thermo Fisher - America\\n3.8   \n",
       "\n",
       "                                           job title       headquarters  \\\n",
       "0                                     Data Scientist      Sunnyvale, CA   \n",
       "1                                     Data Scientist  Mountain View, CA   \n",
       "2  Principal Data Scientist - Candidate Recommend...  San Francisco, CA   \n",
       "3     Senior Data Scientist - Moderation Engineering  San Francisco, CA   \n",
       "4                                 Data Scientist III  San Francisco, CA   \n",
       "\n",
       "                          salary estimate              job type  \\\n",
       "0                                      -1        Job Type : N/A   \n",
       "1                                      -1  Job Type : Full-time   \n",
       "2  Employer Provided Salary:$187K - $231K  Job Type : Full-time   \n",
       "3  Employer Provided Salary:$130K - $156K  Job Type : Full-time   \n",
       "4                                      -1        Job Type : N/A   \n",
       "\n",
       "                    size  founded               type  \\\n",
       "0       10000+ employees     1994   company - public   \n",
       "1  501 to 1000 employees     2016  company - private   \n",
       "2       10000+ employees     2004  company - private   \n",
       "3       10000+ employees     2004  company - private   \n",
       "4       10000+ employees     1902   company - public   \n",
       "\n",
       "                            industry                     sector  \\\n",
       "0  general merchandise & superstores                     retail   \n",
       "1                           internet     information technology   \n",
       "2                           internet     information technology   \n",
       "3                           internet     information technology   \n",
       "4          biotech & pharmaceuticals  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue                                    job description  \n",
       "0        $10+ billion (usd)  Position Summary...\\nWhat you'll do...\\nAnalyt...  \n",
       "1  unknown / non-applicable  TikTok is the leading destination for short-fo...  \n",
       "2    $2 to $5 billion (usd)  Your Job\\nThe Candidate Recommendations team b...  \n",
       "3    $2 to $5 billion (usd)  Your Job\\nThe Moderation Engineering team’s mi...  \n",
       "4        $10+ billion (usd)  Thermo Fisher Scientific Inc. is the world lea...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2573, 12)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['company', 'job title', 'headquarters', 'salary estimate', 'job type',\n",
       "       'size', 'founded', 'type', 'industry', 'sector', 'revenue',\n",
       "       'job description'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>founded</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2573.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1485.608628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>866.857576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1995.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2009.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2019.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           founded\n",
       "count  2573.000000\n",
       "mean   1485.608628\n",
       "std     866.857576\n",
       "min      -1.000000\n",
       "25%      -1.000000\n",
       "50%    1995.000000\n",
       "75%    2009.000000\n",
       "max    2019.000000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observation_univariate: [0, \"founded\"].\\\n",
    " \\\n",
    " All the values for the \"founded\" feature are biased\\\n",
    " because they have a great deal of missing values (-1)\\\n",
    " and this would skew both the mean, std, and the median and quartiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2573 entries, 0 to 2572\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype \n",
      "---  ------           --------------  ----- \n",
      " 0   company          2573 non-null   object\n",
      " 1   job title        2573 non-null   object\n",
      " 2   headquarters     2573 non-null   object\n",
      " 3   salary estimate  2573 non-null   object\n",
      " 4   job type         2573 non-null   object\n",
      " 5   size             2573 non-null   object\n",
      " 6   founded          2573 non-null   int64 \n",
      " 7   type             2573 non-null   object\n",
      " 8   industry         2573 non-null   object\n",
      " 9   sector           2573 non-null   object\n",
      " 10  revenue          2573 non-null   object\n",
      " 11  job description  2573 non-null   object\n",
      "dtypes: int64(1), object(11)\n",
      "memory usage: 261.3+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 misc_obsvtn: [0,].\\\n",
    " \\\n",
    " There are no null or NaN values but they are actually represented as -1s.\\\n",
    " Everything is an object except for the year founded which is int64."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Wrangling and Cleaning the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>job title</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>founded</th>\n",
       "      <th>type</th>\n",
       "      <th>industry</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "      <th>job description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Walmart\\n3.4</td>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>1994</td>\n",
       "      <td>company - public</td>\n",
       "      <td>general merchandise &amp; superstores</td>\n",
       "      <td>retail</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "      <td>Position Summary...\\nWhat you'll do...\\nAnalyt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TikTok\\n3.8</td>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>501 to 1000 employees</td>\n",
       "      <td>2016</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "      <td>TikTok is the leading destination for short-fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>Principal Data Scientist - Candidate Recommend...</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$187K - $231K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>2004</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "      <td>Your Job\\nThe Candidate Recommendations team b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>Senior Data Scientist - Moderation Engineering</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$130K - $156K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>2004</td>\n",
       "      <td>company - private</td>\n",
       "      <td>internet</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "      <td>Your Job\\nThe Moderation Engineering team’s mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America\\n3.8</td>\n",
       "      <td>Data Scientist III</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>1902</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "      <td>Thermo Fisher Scientific Inc. is the world lea...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        company  \\\n",
       "0                  Walmart\\n3.4   \n",
       "1                   TikTok\\n3.8   \n",
       "2                   Indeed\\n4.3   \n",
       "3                   Indeed\\n4.3   \n",
       "4  Thermo Fisher - America\\n3.8   \n",
       "\n",
       "                                           job title       headquarters  \\\n",
       "0                                     Data Scientist      Sunnyvale, CA   \n",
       "1                                     Data Scientist  Mountain View, CA   \n",
       "2  Principal Data Scientist - Candidate Recommend...  San Francisco, CA   \n",
       "3     Senior Data Scientist - Moderation Engineering  San Francisco, CA   \n",
       "4                                 Data Scientist III  San Francisco, CA   \n",
       "\n",
       "                          salary estimate              job type  \\\n",
       "0                                      -1        Job Type : N/A   \n",
       "1                                      -1  Job Type : Full-time   \n",
       "2  Employer Provided Salary:$187K - $231K  Job Type : Full-time   \n",
       "3  Employer Provided Salary:$130K - $156K  Job Type : Full-time   \n",
       "4                                      -1        Job Type : N/A   \n",
       "\n",
       "                    size  founded               type  \\\n",
       "0       10000+ employees     1994   company - public   \n",
       "1  501 to 1000 employees     2016  company - private   \n",
       "2       10000+ employees     2004  company - private   \n",
       "3       10000+ employees     2004  company - private   \n",
       "4       10000+ employees     1902   company - public   \n",
       "\n",
       "                            industry                     sector  \\\n",
       "0  general merchandise & superstores                     retail   \n",
       "1                           internet     information technology   \n",
       "2                           internet     information technology   \n",
       "3                           internet     information technology   \n",
       "4          biotech & pharmaceuticals  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue                                    job description  \n",
       "0        $10+ billion (usd)  Position Summary...\\nWhat you'll do...\\nAnalyt...  \n",
       "1  unknown / non-applicable  TikTok is the leading destination for short-fo...  \n",
       "2    $2 to $5 billion (usd)  Your Job\\nThe Candidate Recommendations team b...  \n",
       "3    $2 to $5 billion (usd)  Your Job\\nThe Moderation Engineering team’s mi...  \n",
       "4        $10+ billion (usd)  Thermo Fisher Scientific Inc. is the world lea...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean = df.copy()\n",
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Webscraping is super messy, especially when it comes to joblistings! I've implemented some general transformations for removing NaN Rows and certain specified columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Remove rows if too many missing values.\n",
    "\n",
    "class RemoveNaNRows(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, cnt=5):\n",
    "        self.cnt = cnt\n",
    "\n",
    "    def fit(self, X, y=None):  \n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        filter_ = X_copy.apply(lambda row: False if Counter(row.values)[\"-1\"] >= self.cnt else True, axis=1)\n",
    "        X_copy = X_copy[filter_]\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)\n",
    "\n",
    "# 1. Select certain columns.\n",
    "\n",
    "class DataFrameSelector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, attribute_names=[], include=True):\n",
    "        self.attribute_names = attribute_names\n",
    "        self.include = include\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        if not self.attribute_names:\n",
    "            return X\n",
    "        \n",
    "        if self.include:\n",
    "            return X[self.attribute_names]\n",
    "        \n",
    "        cols = OrderedSet(X.columns).difference(OrderedSet(self.attribute_names))\n",
    "        return X[cols]\n",
    "        \n",
    "\n",
    "# 1. Remove unnecessary indices at the beginning.\n",
    "\n",
    "class IndexRemover(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "                \n",
    "        unnamed_cols = [col for col in X_copy.columns if \"Unnamed\" in col]\n",
    "        try: X_copy = X_copy.drop(columns=unnamed_cols)\n",
    "        except: pass\n",
    "                                 \n",
    "        return X_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ❓ Questions: 0.\\\n",
    "  \\\n",
    "  Which columns should we be removing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will remove the \"job title\" column because it won't be too useful (all jobs are data science or data science related). \"job description\" is an NLP task which we won't tackle (actually, maybe we can extract key words or frequent words using an NLP method!). Upon inspection, \"founded\" has too many NaNs and \"industry\" is too similar to sector (which has less NaNs) so we simply drop \"industry\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_general_pipeline = Pipeline([\n",
    "    (\"nan_row_remover\", RemoveNaNRows()),\n",
    "    (\"col_selector\", DataFrameSelector([\"job title\", \"founded\", \"job description\", \"industry\"], include=False)),\n",
    "    (\"index_remover\", IndexRemover())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Walmart\\n3.4</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>retail</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TikTok\\n3.8</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>501 to 1000 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$187K - $231K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Indeed\\n4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$130K - $156K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America\\n3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        company       headquarters  \\\n",
       "0                  Walmart\\n3.4      Sunnyvale, CA   \n",
       "1                   TikTok\\n3.8  Mountain View, CA   \n",
       "2                   Indeed\\n4.3  San Francisco, CA   \n",
       "3                   Indeed\\n4.3  San Francisco, CA   \n",
       "4  Thermo Fisher - America\\n3.8  San Francisco, CA   \n",
       "\n",
       "                          salary estimate              job type  \\\n",
       "0                                      -1        Job Type : N/A   \n",
       "1                                      -1  Job Type : Full-time   \n",
       "2  Employer Provided Salary:$187K - $231K  Job Type : Full-time   \n",
       "3  Employer Provided Salary:$130K - $156K  Job Type : Full-time   \n",
       "4                                      -1        Job Type : N/A   \n",
       "\n",
       "                    size               type                     sector  \\\n",
       "0       10000+ employees   company - public                     retail   \n",
       "1  501 to 1000 employees  company - private     information technology   \n",
       "2       10000+ employees  company - private     information technology   \n",
       "3       10000+ employees  company - private     information technology   \n",
       "4       10000+ employees   company - public  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue  \n",
       "0        $10+ billion (usd)  \n",
       "1  unknown / non-applicable  \n",
       "2    $2 to $5 billion (usd)  \n",
       "3    $2 to $5 billion (usd)  \n",
       "4        $10+ billion (usd)  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean = clean_general_pipeline.fit_transform(df_clean)\n",
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1. company and rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Facebook\\n4.3                      119\n",
       "Ursus\\n4.4                          51\n",
       "Uber\\n4.0                           43\n",
       "Intuit - Data\\n4.5                  33\n",
       "Salesforce\\n4.4                     30\n",
       "                                  ... \n",
       "Coherus BioSciences\\n4.5             1\n",
       "Zipline\\n4.3                         1\n",
       "Nova Measuring Instruments\\n3.8      1\n",
       "JGB Biopharma Consulting Inc         1\n",
       "Wild Type\\n5.0                       1\n",
       "Name: company, Length: 770, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Looking at all unique values and their counts for \n",
    "# the \"company\" attribute and sorting by ascending.\n",
    "companies = df[\"company\"].value_counts()\n",
    "companies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ❓ Questions: 1.\\\n",
    "  \\\n",
    "  How many unique companies are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "695"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# My process:\n",
    "# - split ratings out\n",
    "# - apply set() to remove duplicates\n",
    "\n",
    "unique_companies = set([company.split(\"\\n\")[0] for company in companies.index])\n",
    "len(unique_companies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'10x Genomics',\n",
       " '23andMe',\n",
       " '3T Biosciences',\n",
       " '4Insite',\n",
       " '6sense',\n",
       " '8k Miles',\n",
       " 'ABD Insurance and Financial Services',\n",
       " 'ACL Digital',\n",
       " 'ALLDATA',\n",
       " 'ALSTEM',\n",
       " 'AMPAC Fine Chemicals',\n",
       " 'APTIM',\n",
       " 'AT&T',\n",
       " 'AVA ROAD',\n",
       " 'AVANADE',\n",
       " 'Abbott Laboratories',\n",
       " 'Abl Schools',\n",
       " 'Acara Solutions',\n",
       " 'Accenture',\n",
       " 'Accrete Hitech Solutions',\n",
       " 'Acorn Analytics',\n",
       " 'Acumen LLC',\n",
       " 'Adecco',\n",
       " 'Adobe',\n",
       " 'Advanced Systems Group',\n",
       " 'Adventist Health',\n",
       " 'Affimedix Inc',\n",
       " 'Afresh',\n",
       " 'Afterpay Touch',\n",
       " 'Agama Solutions',\n",
       " 'Agilent Technologies, Inc.',\n",
       " 'Aible',\n",
       " 'Akoya Biosciences',\n",
       " 'Akraya Inc.',\n",
       " 'Alameda Health Consortium/Community Health Center Network',\n",
       " 'Alexa Internet',\n",
       " 'AlignTech',\n",
       " 'Alkahest, Inc.',\n",
       " 'Allness Inc',\n",
       " 'Allscripts',\n",
       " 'Alten',\n",
       " 'Alto Neuroscience, Inc.',\n",
       " 'Alvah Contractors, Inc.',\n",
       " 'Amazon Dev Center U.S., Inc.',\n",
       " 'Amazon Web Services, Inc.',\n",
       " 'Amazon.com Services LLC',\n",
       " 'Ambys Medicines',\n",
       " 'Amino, Inc.',\n",
       " 'Amobee',\n",
       " 'AmpersandPeople',\n",
       " 'Anaspec, Inc',\n",
       " 'Anthem',\n",
       " 'Anzu Global',\n",
       " 'Aon Corporation',\n",
       " 'Apexon',\n",
       " 'Apollomics',\n",
       " 'Apostle',\n",
       " 'AppZen',\n",
       " 'Apple',\n",
       " 'Applied Materials Inc.',\n",
       " 'Apton Biosystems',\n",
       " 'Aquabyte',\n",
       " 'Aridis Pharmaceuticals',\n",
       " 'Arkose Labs',\n",
       " 'Artius',\n",
       " 'Aruba Networks',\n",
       " 'Aspire Public Schools',\n",
       " 'Association For Energy Affordability, Inc.',\n",
       " 'Assured',\n",
       " 'Astreya Partners',\n",
       " 'Astrix Technology Group',\n",
       " 'Atlassian',\n",
       " 'Aurora Innovation',\n",
       " 'AutoGrid Systems',\n",
       " 'AutoZone',\n",
       " 'Autodesk',\n",
       " 'Avila Trading',\n",
       " 'BEPC Inc',\n",
       " 'BICP',\n",
       " 'BISHOP WISECARVER CORPORATION',\n",
       " 'BOLD',\n",
       " 'Baidu USA',\n",
       " 'Bay Area Community Health',\n",
       " 'Bayer',\n",
       " 'Bayer AG',\n",
       " 'Beepi',\n",
       " 'Belcan',\n",
       " 'Benchling',\n",
       " 'Beshton Software',\n",
       " 'Bevov',\n",
       " 'Big Data Federation',\n",
       " 'Big Fish Games',\n",
       " 'Bill.com',\n",
       " 'Bio-Rad Laboratories, Inc.',\n",
       " 'BioEclipse Therapeutics, Inc.',\n",
       " 'BioMart Inc',\n",
       " 'Bioethic Consulting',\n",
       " 'BlackRock',\n",
       " 'Bloom Energy',\n",
       " 'Blue Owl',\n",
       " 'Blue River Technology',\n",
       " 'Blue Shield of California',\n",
       " 'Bolt',\n",
       " 'Boomi',\n",
       " 'Braintrust',\n",
       " 'Bristol Myers Squibb',\n",
       " 'C3 AI',\n",
       " 'CA-One Tech Cloud Inc',\n",
       " 'CBRE',\n",
       " 'CBS Interactive',\n",
       " 'CCRM',\n",
       " 'CDI Solutions',\n",
       " 'COPA Soccer Training Center',\n",
       " 'CPP Investments',\n",
       " 'CVPartners',\n",
       " 'CVS Health',\n",
       " 'CaaStle',\n",
       " 'Cadence Design Systems',\n",
       " 'Calico',\n",
       " 'California State University',\n",
       " 'Capella Space',\n",
       " 'Capgemini',\n",
       " 'Capital One - US',\n",
       " 'Carbon Health',\n",
       " 'Carbon Mapper',\n",
       " 'Carbon3D',\n",
       " 'Carmot Therapeutics, Inc.',\n",
       " 'Carnegie Institution',\n",
       " 'Carta',\n",
       " 'Cash App',\n",
       " 'Castlight Health',\n",
       " 'Catalyst Biosciences',\n",
       " 'Cavnue',\n",
       " 'Celandyne Software Solutions',\n",
       " 'Centrify',\n",
       " 'Chan Zuckerberg Biohub',\n",
       " 'Chan Zuckerberg Initiative',\n",
       " 'Change Healthcare',\n",
       " 'Change Research',\n",
       " 'Charles River',\n",
       " 'Charles River Laboratories',\n",
       " 'Checkmate Partners',\n",
       " 'Cheetah Mobile',\n",
       " 'Chelsea Search Group',\n",
       " 'Chime',\n",
       " 'Chinese Hospital',\n",
       " 'CipherTrace',\n",
       " 'CircleCI',\n",
       " 'Circularis Biotechnologies',\n",
       " 'Cisco Systems',\n",
       " 'CitiusTech',\n",
       " 'Clearbit',\n",
       " 'ClimateAI',\n",
       " 'Clinical Management Consultants',\n",
       " 'CloudMedx',\n",
       " 'Cloudflare',\n",
       " 'Codexis',\n",
       " 'Cogitativo',\n",
       " 'Cognizant Technology Solutions',\n",
       " 'Coherus BioSciences',\n",
       " 'Collective Health',\n",
       " 'Colsh Consultants',\n",
       " 'Commonstock',\n",
       " 'Compassion Pathways',\n",
       " 'Compunnel Software Group',\n",
       " 'Concept Tapestry',\n",
       " 'Confluent',\n",
       " 'Contus',\n",
       " 'CooTek',\n",
       " 'Cornershop by Uber',\n",
       " 'Cotocus',\n",
       " 'Couchbase',\n",
       " 'County of Contra Costa',\n",
       " 'County of San Mateo',\n",
       " 'Course Hero',\n",
       " 'Covered California',\n",
       " 'Creative Circle',\n",
       " 'Credible',\n",
       " 'Cree Inc',\n",
       " 'CriticalRiver Technologies Inc.',\n",
       " 'Cruise',\n",
       " 'Cuberg',\n",
       " 'Curai',\n",
       " 'CyberCoders',\n",
       " 'CytomX Therapeutics, Inc.',\n",
       " 'DNA Script',\n",
       " 'DNAlite Therapeutics',\n",
       " 'DNAnexus',\n",
       " 'DTEX Systems',\n",
       " 'DUOPEAK',\n",
       " 'DataVisor',\n",
       " 'Databricks',\n",
       " 'Datatron Technologies',\n",
       " 'Davis Sensory Institute',\n",
       " 'DeepMap, Inc',\n",
       " 'DeepVu',\n",
       " 'Deeproute.ai',\n",
       " 'Del Monte Foods, Inc.',\n",
       " 'Dell Technologies',\n",
       " 'Denali System Co., Ltd',\n",
       " 'Denali Therapeutics',\n",
       " 'DiDi Labs',\n",
       " 'Dictionary.com',\n",
       " 'Diffbot',\n",
       " 'Dimensional Control Systems',\n",
       " 'Discord',\n",
       " 'Divvy Homes',\n",
       " 'DocuSign',\n",
       " 'Dolby',\n",
       " 'DoorDash',\n",
       " 'EA Engineering Science and Technology',\n",
       " 'ELT Solutions',\n",
       " 'EOS',\n",
       " 'EPM Scientific',\n",
       " 'Earli, Inc.',\n",
       " 'Ease',\n",
       " 'Edgesource Corporation',\n",
       " 'Electronic Arts',\n",
       " 'Eluvio',\n",
       " 'Embark Trucks',\n",
       " 'Energy Solutions',\n",
       " 'Enologix',\n",
       " 'Epic',\n",
       " 'Epic Games',\n",
       " 'Equinix',\n",
       " 'Ericsson-Worldwide',\n",
       " 'Eugenus Inc',\n",
       " 'Eurofins Central Laboratory',\n",
       " 'Exponent',\n",
       " 'Extend',\n",
       " 'F5',\n",
       " 'FICO',\n",
       " 'FT Optimize',\n",
       " 'Facebook',\n",
       " 'Faire',\n",
       " 'Fanatics',\n",
       " 'Fanatics Inc.',\n",
       " 'Feedzai',\n",
       " 'FibroGen, Inc.',\n",
       " 'Finrise',\n",
       " 'First Place for Youth',\n",
       " 'First Republic Bank',\n",
       " 'Fiserv',\n",
       " 'Fluence',\n",
       " 'FocusKPI Inc.',\n",
       " 'Food Management Search',\n",
       " 'Ford Motor Company',\n",
       " 'Forethought',\n",
       " 'Forward',\n",
       " 'Fracta',\n",
       " 'Freenome',\n",
       " 'Freenome Holdings',\n",
       " 'Fremont Bank',\n",
       " 'FutureSoft IT',\n",
       " 'Fyusion',\n",
       " 'G4S',\n",
       " 'GA Instructor',\n",
       " 'GRAIL, Inc.',\n",
       " 'GSK',\n",
       " 'GSN Games',\n",
       " 'Gables Search Group',\n",
       " 'Galvanize',\n",
       " 'Galvanize Inc.',\n",
       " 'Gamma Reality Inc.',\n",
       " 'Gap Inc.',\n",
       " \"Gayathri's Sandbox\",\n",
       " 'Geli',\n",
       " 'Genentech',\n",
       " 'Gilead Sciences',\n",
       " 'Glassdoor',\n",
       " 'Global Inc',\n",
       " 'Glu Mobile',\n",
       " 'GokeUSLab',\n",
       " 'Goldman Sachs',\n",
       " 'Goodway Group',\n",
       " 'Google',\n",
       " 'Gotion, Inc.',\n",
       " 'Grabango',\n",
       " 'Grammarly, Inc.',\n",
       " 'Granular',\n",
       " 'Grid Dynamics',\n",
       " 'GroundTruth',\n",
       " 'Grove Collaborative',\n",
       " 'Guidehouse',\n",
       " 'Guidewire Software, Inc.',\n",
       " 'HCL Technologies',\n",
       " 'Happy Elements',\n",
       " 'Harmonize',\n",
       " 'Harnham',\n",
       " 'Harnham US',\n",
       " 'Health Plan of San Mateo',\n",
       " 'Health at Scale Corporation',\n",
       " 'Hearth',\n",
       " 'Heluna Health',\n",
       " 'Hinge Health',\n",
       " 'Hongene Biotech Corporation',\n",
       " 'HopHR',\n",
       " 'Houzz',\n",
       " 'Humanyze',\n",
       " 'HypaHub, Inc.',\n",
       " 'IBM',\n",
       " 'IDEASVOICE',\n",
       " 'Impossible Foods',\n",
       " 'Incept Data Solutions, Inc',\n",
       " 'Indeed',\n",
       " 'Innoplexus',\n",
       " 'Innovyt',\n",
       " 'Instacart - Corporate',\n",
       " 'Intel',\n",
       " 'Intelliswift Software Inc',\n",
       " 'Intelygenz Inc',\n",
       " 'Intercom',\n",
       " 'International Consulting Group Inc.',\n",
       " 'Intuit',\n",
       " 'Intuit - Data',\n",
       " 'Intuitive',\n",
       " 'Iron Ox',\n",
       " 'Iron Ox, Inc.',\n",
       " 'Ironclad, Inc.',\n",
       " 'JD.com',\n",
       " 'JGB Biopharma Consulting Inc',\n",
       " 'JLL',\n",
       " 'JM Talent Inc.',\n",
       " 'January',\n",
       " 'Jazz Pharma',\n",
       " 'Jerry',\n",
       " 'Jobot',\n",
       " 'Joby Aviation',\n",
       " 'Johnson & Johnson Family of Companies',\n",
       " 'Johnson Controls',\n",
       " 'Juniper Networks',\n",
       " 'KJ Consulting LLC',\n",
       " 'KLA-Tencor',\n",
       " 'KLM Careers',\n",
       " 'KPMG',\n",
       " 'Kaiser Permanente',\n",
       " 'Keewi',\n",
       " 'Kelly',\n",
       " 'Kodiak Robotics',\n",
       " 'Komodo Health',\n",
       " 'Kuaishou Technology',\n",
       " 'LG Energy Solution Michigan, Inc.',\n",
       " 'LGC Limited',\n",
       " 'LVIS Corporation',\n",
       " 'LaBine and Associates',\n",
       " 'Laboratory Corporation of America Holding',\n",
       " 'Laguna Source',\n",
       " 'LakePharma, Inc.',\n",
       " 'Lawrence Berkeley National Laboratory',\n",
       " 'Lawrence Livermore National Laboratory',\n",
       " 'Lead5 Media',\n",
       " 'Ledgent Technology',\n",
       " 'LegalZoom',\n",
       " 'Lemonaid Health',\n",
       " 'Levi Strauss',\n",
       " 'Levi Strauss & Co.',\n",
       " 'Lighthouse Lab Services',\n",
       " 'Lily AI',\n",
       " 'Living Carbon',\n",
       " 'Loom',\n",
       " 'LotusFlare, Inc.',\n",
       " 'Lucid Motors',\n",
       " 'Luminostics',\n",
       " 'Lyft',\n",
       " 'Lyra Health',\n",
       " 'MATRIX Resources',\n",
       " 'MIcrovi Biotech',\n",
       " 'MOLOCO',\n",
       " 'MR TALENT LLC',\n",
       " 'MW Partners',\n",
       " 'MadKudu',\n",
       " 'Made Renovation',\n",
       " 'Madison Reed',\n",
       " 'MarinHealth Medical Center',\n",
       " 'Marqeta',\n",
       " 'Mason Tillman Associates',\n",
       " 'MasterClass',\n",
       " 'Match Group',\n",
       " 'Match Relevant',\n",
       " 'Matroid',\n",
       " 'McAfee',\n",
       " \"McDonald's Corporate\",\n",
       " 'Mendel',\n",
       " 'Metromile',\n",
       " 'Microsoft',\n",
       " 'Minaris Regenerative Medicine',\n",
       " 'Mirabilis Design',\n",
       " 'Mitre Corporation',\n",
       " 'MomentRanks',\n",
       " 'Monterey Bay Analytical Services',\n",
       " \"Moody's Corporation\",\n",
       " 'Moveworks',\n",
       " 'Moveworks.ai',\n",
       " 'Mythic-AI',\n",
       " 'N3TWORK',\n",
       " 'NCIRE',\n",
       " 'NGUSoft',\n",
       " 'NVIDIA',\n",
       " 'NVIGEN',\n",
       " 'Nauto',\n",
       " 'NephroSant, Inc.',\n",
       " 'Nestle USA',\n",
       " 'Netflix',\n",
       " 'NeuroTrainer',\n",
       " 'New Relic',\n",
       " 'NewsBreak',\n",
       " 'Next Insurance',\n",
       " 'NextEra Energy',\n",
       " 'NextGen America',\n",
       " 'Nextdoor',\n",
       " 'Niantic',\n",
       " 'Nkarta, Inc.',\n",
       " 'Noodle.ai',\n",
       " 'NotCo',\n",
       " 'Notable Labs',\n",
       " 'Notion',\n",
       " 'Nova Measuring Instruments',\n",
       " 'NovoNutrients',\n",
       " 'Numerdox',\n",
       " 'Nuna',\n",
       " 'Nurix',\n",
       " 'Nuro',\n",
       " 'NutaNXT Technologies',\n",
       " 'Nutcracker Therapeutics',\n",
       " 'NxT Level',\n",
       " 'ONX',\n",
       " 'OPPO US Research Center',\n",
       " 'Oak Valley Hospital District',\n",
       " 'Octave Bioscience',\n",
       " 'Omniscience',\n",
       " 'On-Board Companies',\n",
       " 'One Concern',\n",
       " 'Onto Innovation',\n",
       " 'Onward Search',\n",
       " 'Opendoor',\n",
       " 'Opinion Dynamics',\n",
       " 'Oportun. Inc',\n",
       " 'Opsani',\n",
       " 'Optra Systems',\n",
       " 'Ordr',\n",
       " 'Ordr, Inc.',\n",
       " 'Osaro',\n",
       " 'Ossium Health',\n",
       " 'Oura',\n",
       " 'Oxford Global Resources',\n",
       " 'Oxygen',\n",
       " 'PAX Labs',\n",
       " 'PG&E Corporation',\n",
       " 'Palo Alto Networks',\n",
       " 'Parker Institute for Cancer Immunotherapy',\n",
       " 'Parker Institute of Cancer Immunotherapy (PICI)',\n",
       " 'Pasona N A, Inc.',\n",
       " 'PayJoy',\n",
       " 'PayPal',\n",
       " 'Pendulum.co',\n",
       " 'Pendulum™',\n",
       " 'Penumbra US',\n",
       " 'Perfect Minds',\n",
       " 'PerformanceStar',\n",
       " 'Personalis, Inc',\n",
       " 'Picarro',\n",
       " 'PicnicHealth',\n",
       " 'Pinterest',\n",
       " 'Pixalate, Inc.',\n",
       " 'Pixelberry Studios',\n",
       " 'Plaid',\n",
       " 'Platinum Staffing Solutions',\n",
       " 'Plato',\n",
       " 'Plenty',\n",
       " 'Plum Lending',\n",
       " 'Plus',\n",
       " 'Pngme',\n",
       " 'Pocket Worlds',\n",
       " 'Point',\n",
       " 'Poshmark',\n",
       " 'Power Integrations',\n",
       " 'Primary Talent Partners',\n",
       " 'PrimeNeuro',\n",
       " 'Proclinical Staffing',\n",
       " 'Procyon technostructure',\n",
       " 'Proofpoint',\n",
       " 'PubMatic',\n",
       " 'PubNub',\n",
       " 'Punchh',\n",
       " 'Pyramid Consulting, Inc',\n",
       " 'Qualpro Systems',\n",
       " 'Qualys',\n",
       " 'Quantifind',\n",
       " 'Quintara Discovery, Inc (QDI)',\n",
       " 'RAPP',\n",
       " 'REDICA Systems',\n",
       " 'REGID International',\n",
       " 'RELX Group',\n",
       " 'Randstad General Staffing',\n",
       " 'Raybeam, Inc.',\n",
       " 'ReCode Therapeutics',\n",
       " 'Riiid Labs',\n",
       " 'Rivian Automotive',\n",
       " 'Robby Technologies',\n",
       " 'Robert Half Corporate Internal',\n",
       " 'Robinhood',\n",
       " 'Roblox',\n",
       " 'Roche',\n",
       " 'Rubicon Temps',\n",
       " 'Runway Innovation Hub',\n",
       " 'Ryder System',\n",
       " 'SACC INC',\n",
       " 'SADA',\n",
       " 'SKT Lab',\n",
       " 'SRA OSS',\n",
       " 'SRG',\n",
       " 'SRS Consultancy Inc',\n",
       " 'STAR Labs',\n",
       " 'SafeTraces, Inc.',\n",
       " 'Sail Internet',\n",
       " 'Salesforce',\n",
       " 'Samsung Electronics America',\n",
       " 'Samsung SDS America',\n",
       " 'ScaleMonk',\n",
       " 'Seasoned, LLC',\n",
       " 'Second Genome',\n",
       " 'Secureworks',\n",
       " 'Seer',\n",
       " 'Sensor Tower',\n",
       " 'Sentient Energy',\n",
       " 'SeqMatic, LLC',\n",
       " 'Shoe Palace Corporation',\n",
       " 'Sightline Payments',\n",
       " 'Sigma Inc',\n",
       " 'Sigmaways, Inc.',\n",
       " 'Sikka Software',\n",
       " 'SiriusXM',\n",
       " 'Sisu',\n",
       " 'Sitecore',\n",
       " 'Skan',\n",
       " 'Slack',\n",
       " 'Slalom LLC.',\n",
       " 'Snapchat',\n",
       " 'Snark AI',\n",
       " 'SoFi',\n",
       " 'Solactron',\n",
       " 'Sony Corporation of America',\n",
       " 'Sony Interactive Entertainment PlayStation',\n",
       " 'Sony PlayStation',\n",
       " 'SpanIdea',\n",
       " 'Spartan Technologies',\n",
       " 'Specific Diagnostics',\n",
       " 'Spin',\n",
       " 'Splunk',\n",
       " 'Spring Discovery',\n",
       " 'Squadex',\n",
       " 'Square',\n",
       " 'Stanford University',\n",
       " 'Stealth Mode Startup',\n",
       " 'Stealth Mode Startup Company',\n",
       " 'Step',\n",
       " 'Step Mobile',\n",
       " 'Stitch Fix',\n",
       " 'Store No. 8',\n",
       " 'Strategi.biz',\n",
       " 'StubHub',\n",
       " 'Sunshine',\n",
       " 'SupportLogic',\n",
       " 'Surya Systems',\n",
       " 'Susquehanna International Group',\n",
       " 'Sutro Biopharma',\n",
       " 'SymphonyAI',\n",
       " 'Synapse Financial Technologies Inc',\n",
       " 'Synopsys',\n",
       " 'TCL Research America',\n",
       " 'TEKPLAY SYSTEMS',\n",
       " 'TEKsystems',\n",
       " 'TSN Partners',\n",
       " 'Tailored Management',\n",
       " 'Tapjoy',\n",
       " 'TargetCW',\n",
       " 'TaskRabbit',\n",
       " 'TechTrueUP',\n",
       " 'Technosoft Global',\n",
       " 'Teknatio',\n",
       " 'Tekrek Solutions',\n",
       " 'Tekruiting',\n",
       " 'Teladoc Health',\n",
       " 'Tempus',\n",
       " 'Tempus Labs',\n",
       " 'Tesla',\n",
       " 'The Beans',\n",
       " 'The Climate Corporation',\n",
       " 'The Mom Project',\n",
       " 'The Rockridge Group',\n",
       " 'The Stone Research Foundation',\n",
       " 'The Voleon Group',\n",
       " 'Therini Bio, Inc.',\n",
       " 'Thermo Fisher - America',\n",
       " 'Thermo Fisher Scientific',\n",
       " 'ThirdLove',\n",
       " 'Tiger Analytics',\n",
       " 'TikTok',\n",
       " 'Tinder',\n",
       " 'Toyota Research Institute',\n",
       " 'Treasure Island Sailing Center',\n",
       " 'TrueBinding',\n",
       " 'TuSimple, Inc.',\n",
       " 'Tubi',\n",
       " 'Turo',\n",
       " 'Twilio',\n",
       " 'Twist Bioscience',\n",
       " 'Twitch',\n",
       " 'Twitch Interactive, Inc.',\n",
       " 'Twitter',\n",
       " 'UC Hastings',\n",
       " 'UC San Francisco Academic',\n",
       " 'Uber',\n",
       " 'Udemy',\n",
       " 'Ultimate Staffing Services',\n",
       " 'UnifyID',\n",
       " 'Uniphore',\n",
       " 'United One Solutions',\n",
       " 'Universities Space Research Association',\n",
       " 'University of California Berkeley',\n",
       " 'University of California San Francisco',\n",
       " 'University of California Santa Cruz',\n",
       " 'University of California, Davis',\n",
       " 'Unlearn.AI',\n",
       " 'Unstoppable Domains',\n",
       " 'Upstart',\n",
       " 'Upstart Network, Inc.',\n",
       " 'Ursus',\n",
       " 'UserLeap',\n",
       " 'V Projects',\n",
       " 'V-Soft Consulting Group, Inc.',\n",
       " 'VIZIO, Inc.',\n",
       " 'VMware',\n",
       " 'VTEK SYSTEMS',\n",
       " 'Varo Bank',\n",
       " 'Varo Money',\n",
       " 'Vectra',\n",
       " 'Veeva Systems',\n",
       " 'Veracyte',\n",
       " 'Verana Health',\n",
       " 'Verb Surgical',\n",
       " 'Verkada',\n",
       " 'VertiSystem Inc',\n",
       " 'Viaduct',\n",
       " 'Visual Concepts',\n",
       " 'Vitria Technology',\n",
       " 'Vungle',\n",
       " 'Walmart',\n",
       " 'Walrus Health',\n",
       " \"Warman O'Brien\",\n",
       " 'Waymo',\n",
       " 'Waynsys',\n",
       " 'WeDriveU, Inc.',\n",
       " 'Wealthfront',\n",
       " 'Webhelp Americas',\n",
       " 'Wells Fargo',\n",
       " 'Wells Fargo Bank',\n",
       " 'WhoWhatWhy',\n",
       " 'Wider Circle',\n",
       " 'Wild Type',\n",
       " 'Williams Sonoma',\n",
       " 'Wing',\n",
       " 'Wish',\n",
       " 'Woodruff Sawyer',\n",
       " 'WorkWhile',\n",
       " 'Workday',\n",
       " 'Workforce Logiq',\n",
       " 'Wyzant',\n",
       " 'Xcel Agency',\n",
       " 'Xerox',\n",
       " 'Xoriant Corporation',\n",
       " 'Yardzen',\n",
       " 'ZS Associates',\n",
       " 'Zeta Global',\n",
       " 'Zipcar, Inc.',\n",
       " 'Zipline',\n",
       " 'Zodiac Solutions',\n",
       " 'Zscaler',\n",
       " 'Zymergen',\n",
       " 'a.k.a. Brands',\n",
       " 'centraprise',\n",
       " 'dunnhumby',\n",
       " 'eBay',\n",
       " 'eBay Inc.',\n",
       " 'eHealth',\n",
       " 'f5',\n",
       " 'financial services company',\n",
       " 'freshworks',\n",
       " 'iFoodDecisionSciences',\n",
       " 'infolob',\n",
       " 'information technology sector',\n",
       " 'kover',\n",
       " 'test.ai',\n",
       " 'tunein',\n",
       " 'worldcoin.org'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_companies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"company\"].\\\n",
    "  \\\n",
    "  695 unique companies, but looks like some overlap like \"Wells Fargo\" and \"Wells Fargo Bank\"\\\n",
    "  and some other companies that aren't company names like \"information technology sector\" or \"financial services company\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ❓ Questions: 2.\\\n",
    "  \\\n",
    "  What percentage of instances don't have ratings?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "About 13.6% of the dataset doesn't have ratings.\n"
     ]
    }
   ],
   "source": [
    "p = len([c for c in df.company if \"\\n\" not in c])/len(df) * 100\n",
    "\n",
    "print(f\"About {p:.1f}% of the dataset doesn't have ratings.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [1, \"company\"].\\\n",
    "  \\\n",
    "  13.6% is a good chunk of the data, but my threshold for ignoring possible features\\\n",
    "  is 15%. I think it might be useful to include rating in the dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"company\" column still needs a lot more cleaning. Namely, we need to identify same companies and replace them with 1 name. This will make it easier to group them later on.\n",
    "\n",
    "We can use some type of simple NLP approach to identify how similar 2 strings are, but that runs the risk of mislabeling data. And even then, if we compared each word with all other words, that's $O(N^2)$ comparisons!\n",
    "\n",
    "Since we only have 695 unique companies, I'll simply hardcode rules to cover the edge cases seen so far. Though generally for more complex tasks like with a consistent stream of new data, we might want to shift to a more generalizeable setup. On the client-side, data entry options can be enforced (though not really in this case) and on our side we can either build a really robust parser or build a bot/AI that can compare semantic meaning between words to detect similar company names that are likely to be the same company. This can leverage unsupervised learning methods! But, for now we will stick with a simple hardcoded parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'10x Genomics\\n4.1',\n",
       " '23andMe\\n4.1',\n",
       " '3T Biosciences',\n",
       " '4Insite\\n4.0',\n",
       " '6sense\\n4.9',\n",
       " '8k Miles\\n3.9',\n",
       " 'ABD Insurance and Financial Services\\n4.7',\n",
       " 'ACL Digital\\n3.5',\n",
       " 'ACL Digital\\n3.6',\n",
       " 'ALLDATA\\n4.1',\n",
       " 'ALSTEM\\n3.7',\n",
       " 'AMPAC Fine Chemicals\\n3.1',\n",
       " 'APTIM\\n3.4',\n",
       " 'AT&T\\n3.7',\n",
       " 'AVANADE\\n4.1',\n",
       " 'Abbott Laboratories\\n3.9',\n",
       " 'Abl Schools',\n",
       " 'Acara Solutions\\n3.5',\n",
       " 'Accenture\\n4.0',\n",
       " 'Accrete Hitech Solutions',\n",
       " 'Acorn Analytics',\n",
       " 'Acumen LLC\\n3.3',\n",
       " 'Adecco\\n3.7',\n",
       " 'Adobe\\n4.4',\n",
       " 'Advanced Systems Group\\n3.0',\n",
       " 'Adventist Health\\n3.6',\n",
       " 'Afresh\\n5.0',\n",
       " 'Afterpay Touch\\n3.7',\n",
       " 'Agama Solutions\\n3.7',\n",
       " 'Agilent Technologies, Inc.\\n4.3',\n",
       " 'Aible\\n5.0',\n",
       " 'Akoya Biosciences\\n4.0',\n",
       " 'Akraya Inc.\\n4.6',\n",
       " 'Akraya Inc.\\n4.7',\n",
       " 'Alexa Internet\\n3.8',\n",
       " 'AlignTech\\n4.4',\n",
       " 'Allscripts\\n3.7',\n",
       " 'Alten\\n3.2',\n",
       " 'Alto Neuroscience, Inc.',\n",
       " 'Amazon Dev Center U.S., Inc.\\n3.8',\n",
       " 'Amazon Web Services, Inc.\\n3.8',\n",
       " 'Amazon.com Services LLC\\n3.8',\n",
       " 'Amino, Inc.\\n4.5',\n",
       " 'Amobee\\n3.9',\n",
       " 'Anthem\\n3.6',\n",
       " 'Anzu Global\\n4.5',\n",
       " 'Aon Corporation\\n3.8',\n",
       " 'Apexon\\n4.3',\n",
       " 'Apexon\\n4.4',\n",
       " 'Apostle',\n",
       " 'AppZen\\n3.9',\n",
       " 'Apple\\n4.3',\n",
       " 'Applied Materials Inc.\\n4.0',\n",
       " 'Apton Biosystems',\n",
       " 'Aquabyte\\n3.9',\n",
       " 'Aquabyte\\n4.2',\n",
       " 'Arkose Labs\\n4.4',\n",
       " 'Aruba Networks\\n4.0',\n",
       " 'Aruba Networks\\n4.2',\n",
       " 'Aspire Public Schools\\n3.8',\n",
       " 'Association For Energy Affordability, Inc.\\n2.8',\n",
       " 'Assured',\n",
       " 'Astreya Partners\\n3.7',\n",
       " 'Astrix Technology Group\\n4.2',\n",
       " 'Atlassian\\n4.6',\n",
       " 'Aurora Innovation\\n4.0',\n",
       " 'AutoGrid Systems\\n3.4',\n",
       " 'AutoGrid Systems\\n3.5',\n",
       " 'AutoZone\\n3.3',\n",
       " 'Autodesk\\n4.3',\n",
       " 'Avila Trading\\n3.8',\n",
       " 'Avila Trading\\n3.9',\n",
       " 'BEPC Inc\\n3.7',\n",
       " 'BICP\\n5.0',\n",
       " 'BISHOP WISECARVER CORPORATION\\n3.5',\n",
       " 'BOLD\\n4.1',\n",
       " 'Baidu USA\\n3.9',\n",
       " 'Bay Area Community Health\\n3.0',\n",
       " 'Bayer\\n4.1',\n",
       " 'Bayer\\n4.2',\n",
       " 'Bayer AG\\n4.3',\n",
       " 'Belcan\\n3.6',\n",
       " 'Benchling\\n5.0',\n",
       " 'Beshton Software\\n3.7',\n",
       " 'Bevov\\n4.0',\n",
       " 'Big Data Federation',\n",
       " 'Big Fish Games\\n3.2',\n",
       " 'Big Fish Games\\n3.3',\n",
       " 'Bill.com\\n4.2',\n",
       " 'Bio-Rad Laboratories, Inc.\\n3.5',\n",
       " 'BioMart Inc',\n",
       " 'BlackRock\\n4.1',\n",
       " 'Bloom Energy\\n3.8',\n",
       " 'Blue Owl\\n4.8',\n",
       " 'Blue River Technology\\n4.5',\n",
       " 'Blue Shield of California\\n4.3',\n",
       " 'Blue Shield of California\\n4.4',\n",
       " 'Bolt\\n4.5',\n",
       " 'Bolt\\n4.6',\n",
       " 'Boomi\\n3.8',\n",
       " 'Braintrust\\n5.0',\n",
       " 'Bristol Myers Squibb\\n4.1',\n",
       " 'C3 AI\\n4.6',\n",
       " 'CA-One Tech Cloud Inc\\n5.0',\n",
       " 'CBRE\\n3.9',\n",
       " 'CBS Interactive\\n4.0',\n",
       " 'CCRM\\n3.8',\n",
       " 'CDI Solutions\\n3.8',\n",
       " 'CPP Investments\\n3.6',\n",
       " 'CPP Investments\\n3.7',\n",
       " 'CVPartners\\n4.1',\n",
       " 'CVS Health\\n3.1',\n",
       " 'CaaStle\\n3.5',\n",
       " 'Cadence Design Systems\\n4.4',\n",
       " 'Calico\\n4.2',\n",
       " 'California State University\\n4.3',\n",
       " 'Capella Space\\n4.6',\n",
       " 'Capgemini\\n3.8',\n",
       " 'Capital One - US\\n4.1',\n",
       " 'Carbon Health\\n4.2',\n",
       " 'Carbon3D\\n3.7',\n",
       " 'Carnegie Institution\\n4.3',\n",
       " 'Carta\\n3.9',\n",
       " 'Carta\\n4.0',\n",
       " 'Cash App\\n4.7',\n",
       " 'Cash App\\n4.8',\n",
       " 'Castlight Health\\n4.3',\n",
       " 'Catalyst Biosciences\\n4.2',\n",
       " 'Celandyne Software Solutions',\n",
       " 'Centrify\\n3.4',\n",
       " 'Centrify\\n3.6',\n",
       " 'Chan Zuckerberg Biohub\\n3.4',\n",
       " 'Chan Zuckerberg Initiative\\n4.0',\n",
       " 'Change Healthcare\\n3.4',\n",
       " 'Change Research\\n3.0',\n",
       " 'Charles River Laboratories\\n3.5',\n",
       " 'Cheetah Mobile\\n3.0',\n",
       " 'Chelsea Search Group',\n",
       " 'Chime\\n4.4',\n",
       " 'Chime\\n4.5',\n",
       " 'Chinese Hospital\\n3.8',\n",
       " 'CipherTrace\\n4.2',\n",
       " 'CircleCI\\n3.7',\n",
       " 'CircleCI\\n3.8',\n",
       " 'Cisco Systems\\n4.3',\n",
       " 'CitiusTech\\n3.8',\n",
       " 'Clearbit\\n4.3',\n",
       " 'ClimateAI',\n",
       " 'Clinical Management Consultants\\n3.5',\n",
       " 'Cloudflare\\n4.1',\n",
       " 'Codexis\\n3.2',\n",
       " 'Cogitativo\\n5.0',\n",
       " 'Cognizant Technology Solutions\\n3.8',\n",
       " 'Coherus BioSciences\\n4.4',\n",
       " 'Coherus BioSciences\\n4.5',\n",
       " 'Collective Health\\n3.8',\n",
       " 'Colsh Consultants\\n4.6',\n",
       " 'Commonstock',\n",
       " 'Compassion Pathways',\n",
       " 'Compunnel Software Group\\n3.8',\n",
       " 'Confluent\\n4.5',\n",
       " 'Confluent\\n4.6',\n",
       " 'Contus\\n4.2',\n",
       " 'CooTek\\n4.2',\n",
       " 'Cornershop by Uber\\n4.4',\n",
       " 'Cotocus',\n",
       " 'Couchbase\\n4.8',\n",
       " 'County of Contra Costa\\n4.0',\n",
       " 'County of San Mateo\\n4.0',\n",
       " 'Course Hero\\n4.8',\n",
       " 'Covered California\\n3.8',\n",
       " 'Creative Circle\\n3.6',\n",
       " 'Creative Circle\\n3.7',\n",
       " 'Credible\\n4.3',\n",
       " 'Cree Inc\\n3.4',\n",
       " 'CriticalRiver Technologies Inc.\\n4.6',\n",
       " 'Cruise\\n3.6',\n",
       " 'Cuberg\\n4.7',\n",
       " 'Curai\\n3.9',\n",
       " 'CyberCoders\\n4.0',\n",
       " 'CyberCoders\\n4.1',\n",
       " 'CytomX Therapeutics, Inc.\\n3.4',\n",
       " 'DNA Script\\n5.0',\n",
       " 'DNAnexus\\n4.7',\n",
       " 'DTEX Systems\\n3.6',\n",
       " 'DataVisor\\n4.5',\n",
       " 'Databricks\\n4.8',\n",
       " 'Datatron Technologies\\n4.2',\n",
       " 'DeepMap, Inc\\n4.5',\n",
       " 'Deeproute.ai\\n5.0',\n",
       " 'Del Monte Foods, Inc.\\n3.4',\n",
       " 'Dell Technologies\\n4.2',\n",
       " 'Denali System Co., Ltd\\n3.0',\n",
       " 'Denali Therapeutics\\n3.7',\n",
       " 'DiDi Labs\\n3.7',\n",
       " 'Dictionary.com\\n3.0',\n",
       " 'Diffbot\\n4.6',\n",
       " 'Dimensional Control Systems\\n3.8',\n",
       " 'Discord\\n4.8',\n",
       " 'Divvy Homes\\n3.8',\n",
       " 'DocuSign\\n4.6',\n",
       " 'Dolby\\n4.3',\n",
       " 'DoorDash\\n3.9',\n",
       " 'DoorDash\\n4.0',\n",
       " 'ELT Solutions',\n",
       " 'EOS\\n3.5',\n",
       " 'EOS\\n3.6',\n",
       " 'EOS\\n3.7',\n",
       " 'EPM Scientific\\n2.3',\n",
       " 'Earli, Inc.\\n3.2',\n",
       " 'Ease\\n3.4',\n",
       " 'Edgesource Corporation\\n3.9',\n",
       " 'Electronic Arts\\n4.1',\n",
       " 'Eluvio\\n5.0',\n",
       " 'Embark Trucks\\n4.4',\n",
       " 'Energy Solutions\\n3.7',\n",
       " 'Epic\\n4.7',\n",
       " 'Epic Games\\n4.2',\n",
       " 'Equinix\\n4.4',\n",
       " 'Ericsson-Worldwide\\n4.0',\n",
       " 'Eugenus Inc\\n2.0',\n",
       " 'Eurofins Central Laboratory\\n3.3',\n",
       " 'Exponent\\n3.6',\n",
       " 'Exponent\\n3.8',\n",
       " 'Extend\\n4.6',\n",
       " 'F5\\n4.1',\n",
       " 'FICO\\n3.9',\n",
       " 'FT Optimize\\n5.0',\n",
       " 'Facebook\\n4.3',\n",
       " 'Faire\\n4.7',\n",
       " 'Fanatics\\n3.9',\n",
       " 'Fanatics Inc.\\n3.9',\n",
       " 'Feedzai\\n4.0',\n",
       " 'Feedzai\\n4.1',\n",
       " 'FibroGen, Inc.\\n3.7',\n",
       " 'Finrise\\n5.0',\n",
       " 'First Place for Youth\\n3.2',\n",
       " 'First Republic Bank\\n4.3',\n",
       " 'Fiserv\\n3.3',\n",
       " 'Fiserv\\n3.4',\n",
       " 'Fluence\\n2.5',\n",
       " 'FocusKPI Inc.\\n3.3',\n",
       " 'Food Management Search',\n",
       " 'Ford Motor Company\\n4.1',\n",
       " 'Forethought\\n5.0',\n",
       " 'Forward\\n4.5',\n",
       " 'Fracta\\n4.5',\n",
       " 'Freenome\\n4.9',\n",
       " 'Freenome Holdings\\n4.9',\n",
       " 'Fremont Bank\\n3.0',\n",
       " 'FutureSoft IT\\n5.0',\n",
       " 'Fyusion\\n3.6',\n",
       " 'GRAIL, Inc.\\n3.4',\n",
       " 'GSK\\n3.8',\n",
       " 'GSN Games\\n3.3',\n",
       " 'Gables Search Group\\n3.9',\n",
       " 'Galvanize\\n4.6',\n",
       " 'Galvanize Inc.\\n4.5',\n",
       " 'Gap Inc.\\n3.7',\n",
       " 'Geli\\n3.4',\n",
       " 'Genentech\\n4.3',\n",
       " 'Gilead Sciences\\n3.8',\n",
       " 'Glassdoor\\n3.8',\n",
       " 'Global Inc\\n3.8',\n",
       " 'Glu Mobile\\n4.2',\n",
       " 'GokeUSLab\\n1.3',\n",
       " 'Goldman Sachs\\n4.0',\n",
       " 'Goodway Group\\n4.5',\n",
       " 'Google\\n4.5',\n",
       " 'Gotion, Inc.\\n4.1',\n",
       " 'Grabango\\n3.7',\n",
       " 'Grammarly, Inc.\\n5.0',\n",
       " 'Granular\\n3.5',\n",
       " 'Granular\\n3.6',\n",
       " 'Grid Dynamics\\n4.3',\n",
       " 'GroundTruth\\n3.8',\n",
       " 'Grove Collaborative\\n3.4',\n",
       " 'Guidehouse\\n3.8',\n",
       " 'Guidewire Software, Inc.\\n4.4',\n",
       " 'HCL Technologies\\n3.7',\n",
       " 'Happy Elements\\n4.1',\n",
       " 'Harmonize\\n3.9',\n",
       " 'Harnham\\n3.8',\n",
       " 'Harnham US\\n3.8',\n",
       " 'Health Plan of San Mateo\\n3.8',\n",
       " 'Health at Scale Corporation',\n",
       " 'Hearth\\n2.9',\n",
       " 'Heluna Health\\n3.4',\n",
       " 'Hinge Health\\n4.6',\n",
       " 'HopHR',\n",
       " 'Houzz\\n4.0',\n",
       " 'Humanyze\\n3.5',\n",
       " 'IBM\\n3.9',\n",
       " 'IDEASVOICE\\n4.0',\n",
       " 'Impossible Foods\\n4.3',\n",
       " 'Incept Data Solutions, Inc\\n5.0',\n",
       " 'Indeed\\n4.3',\n",
       " 'Innoplexus\\n3.7',\n",
       " 'Innoplexus\\n3.9',\n",
       " 'Innovyt',\n",
       " 'Instacart - Corporate\\n3.8',\n",
       " 'Intel\\n4.1',\n",
       " 'Intel\\n4.2',\n",
       " 'Intelliswift Software Inc\\n3.8',\n",
       " 'Intelygenz Inc\\n3.6',\n",
       " 'Intelygenz Inc\\n3.7',\n",
       " 'Intercom\\n2.0',\n",
       " 'International Consulting Group Inc.',\n",
       " 'Intuit\\n4.5',\n",
       " 'Intuit - Data\\n4.5',\n",
       " 'Intuitive\\n4.6',\n",
       " 'Iron Ox',\n",
       " 'Iron Ox, Inc.',\n",
       " 'Ironclad, Inc.\\n4.7',\n",
       " 'JD.com\\n3.6',\n",
       " 'JLL\\n3.9',\n",
       " 'JM Talent Inc.',\n",
       " 'January',\n",
       " 'Jazz Pharma\\n4.1',\n",
       " 'Jerry\\n3.4',\n",
       " 'Jobot\\n4.8',\n",
       " 'Joby Aviation\\n4.2',\n",
       " 'Johnson & Johnson Family of Companies\\n4.2',\n",
       " 'Johnson Controls\\n3.5',\n",
       " 'Juniper Networks\\n3.9',\n",
       " 'KLA-Tencor\\n3.9',\n",
       " 'KPMG\\n3.9',\n",
       " 'Kaiser Permanente\\n4.0',\n",
       " 'Kelly\\n3.5',\n",
       " 'Kodiak Robotics\\n4.5',\n",
       " 'Komodo Health\\n4.4',\n",
       " 'Komodo Health\\n4.5',\n",
       " 'Kuaishou Technology\\n4.1',\n",
       " 'LG Energy Solution Michigan, Inc.\\n2.7',\n",
       " 'LGC Limited\\n3.5',\n",
       " 'LGC Limited\\n3.6',\n",
       " 'LVIS Corporation',\n",
       " 'LaBine and Associates\\n5.0',\n",
       " 'Laboratory Corporation of America Holding\\n3.3',\n",
       " 'Laguna Source',\n",
       " 'LakePharma, Inc.\\n2.5',\n",
       " 'Lawrence Berkeley National Laboratory\\n4.3',\n",
       " 'Lawrence Livermore National Laboratory\\n4.6',\n",
       " 'Lead5 Media\\n4.0',\n",
       " 'Ledgent Technology\\n4.1',\n",
       " 'LegalZoom\\n3.2',\n",
       " 'Lemonaid Health\\n4.9',\n",
       " 'Levi Strauss\\n4.1',\n",
       " 'Levi Strauss & Co.\\n4.1',\n",
       " 'Lighthouse Lab Services',\n",
       " 'Lily AI\\n4.5',\n",
       " 'Loom\\n4.9',\n",
       " 'LotusFlare, Inc.\\n4.3',\n",
       " 'Lucid Motors\\n4.1',\n",
       " 'Luminostics',\n",
       " 'Lyft\\n3.9',\n",
       " 'Lyra Health\\n4.6',\n",
       " 'Lyra Health\\n4.8',\n",
       " 'MATRIX Resources\\n4.3',\n",
       " 'MIcrovi Biotech\\n4.2',\n",
       " 'MOLOCO\\n4.1',\n",
       " 'MOLOCO\\n4.2',\n",
       " 'MW Partners\\n4.0',\n",
       " 'MadKudu\\n5.0',\n",
       " 'Made Renovation\\n3.3',\n",
       " 'Madison Reed\\n4.0',\n",
       " 'MarinHealth Medical Center\\n3.4',\n",
       " 'Marqeta\\n4.5',\n",
       " 'Mason Tillman Associates\\n2.7',\n",
       " 'MasterClass\\n4.5',\n",
       " 'Match Group\\n4.0',\n",
       " 'Match Relevant\\n4.3',\n",
       " 'Matroid\\n5.0',\n",
       " 'McAfee\\n3.6',\n",
       " \"McDonald's Corporate\\n3.5\",\n",
       " 'Mendel\\n2.4',\n",
       " 'Metromile\\n4.2',\n",
       " 'Metromile\\n4.3',\n",
       " 'Microsoft\\n4.4',\n",
       " 'Minaris Regenerative Medicine\\n3.1',\n",
       " 'Mirabilis Design\\n4.7',\n",
       " 'Mitre Corporation\\n3.8',\n",
       " 'Monterey Bay Analytical Services',\n",
       " \"Moody's Corporation\\n3.8\",\n",
       " 'Moveworks\\n4.8',\n",
       " 'Moveworks.ai\\n4.8',\n",
       " 'Mythic-AI\\n4.5',\n",
       " 'N3TWORK\\n5.0',\n",
       " 'NCIRE\\n3.4',\n",
       " 'NGUSoft\\n4.4',\n",
       " 'NVIDIA\\n4.7',\n",
       " 'NVIGEN\\n3.0',\n",
       " 'Nauto\\n4.1',\n",
       " 'NephroSant, Inc.',\n",
       " 'Nestle USA\\n4.0',\n",
       " 'Nestle USA\\n4.1',\n",
       " 'Netflix\\n4.1',\n",
       " 'NeuroTrainer',\n",
       " 'New Relic\\n3.9',\n",
       " 'Next Insurance\\n4.7',\n",
       " 'NextEra Energy\\n3.8',\n",
       " 'NextGen America\\n3.7',\n",
       " 'Nextdoor\\n4.6',\n",
       " 'Niantic\\n3.5',\n",
       " 'Niantic\\n3.6',\n",
       " 'Nkarta, Inc.\\n5.0',\n",
       " 'Noodle.ai\\n3.8',\n",
       " 'NotCo\\n3.6',\n",
       " 'Notable Labs\\n5.0',\n",
       " 'Notion\\n4.8',\n",
       " 'Nova Measuring Instruments\\n3.8',\n",
       " 'Nuna\\n4.0',\n",
       " 'Nurix\\n4.0',\n",
       " 'Nuro\\n4.2',\n",
       " 'NutaNXT Technologies\\n3.8',\n",
       " 'NutaNXT Technologies\\n4.0',\n",
       " 'Nutcracker Therapeutics\\n5.0',\n",
       " 'NxT Level\\n5.0',\n",
       " 'ONX\\n4.3',\n",
       " 'OPPO US Research Center\\n4.5',\n",
       " 'Oak Valley Hospital District\\n3.3',\n",
       " 'Omniscience\\n2.3',\n",
       " 'On-Board Companies\\n3.4',\n",
       " 'One Concern\\n3.7',\n",
       " 'Onto Innovation\\n3.6',\n",
       " 'Onward Search\\n3.8',\n",
       " 'Opendoor\\n3.8',\n",
       " 'Oportun. Inc\\n4.2',\n",
       " 'Opsani\\n5.0',\n",
       " 'Optra Systems\\n2.9',\n",
       " 'Ordr',\n",
       " 'Ordr, Inc.\\n4.5',\n",
       " 'Osaro\\n4.8',\n",
       " 'Ossium Health\\n5.0',\n",
       " 'Oura\\n4.9',\n",
       " 'Oxford Global Resources\\n3.6',\n",
       " 'Oxygen\\n3.5',\n",
       " 'PAX Labs\\n2.8',\n",
       " 'PG&E Corporation\\n3.8',\n",
       " 'Palo Alto Networks\\n3.9',\n",
       " 'Parker Institute for Cancer Immunotherapy\\n3.7',\n",
       " 'Parker Institute for Cancer Immunotherapy\\n4.0',\n",
       " 'Pasona N A, Inc.\\n3.8',\n",
       " 'PayJoy\\n4.6',\n",
       " 'PayPal\\n4.1',\n",
       " 'Pendulum.co\\n4.3',\n",
       " 'Penumbra US\\n3.9',\n",
       " 'PerformanceStar\\n2.0',\n",
       " 'Personalis, Inc\\n3.8',\n",
       " 'Personalis, Inc\\n3.9',\n",
       " 'Picarro\\n3.0',\n",
       " 'PicnicHealth\\n4.7',\n",
       " 'Pinterest\\n4.2',\n",
       " 'Pinterest\\n4.3',\n",
       " 'Pixalate, Inc.\\n3.5',\n",
       " 'Pixelberry Studios\\n4.5',\n",
       " 'Plaid\\n4.0',\n",
       " 'Plaid\\n4.6',\n",
       " 'Plato',\n",
       " 'Plenty\\n3.6',\n",
       " 'Plenty\\n3.7',\n",
       " 'Plum Lending\\n4.1',\n",
       " 'Plus\\n5.0',\n",
       " 'Pocket Worlds\\n4.6',\n",
       " 'Point\\n4.9',\n",
       " 'Poshmark\\n4.6',\n",
       " 'Power Integrations\\n3.8',\n",
       " 'Primary Talent Partners\\n3.9',\n",
       " 'Proclinical Staffing\\n4.2',\n",
       " 'Procyon technostructure\\n4.0',\n",
       " 'Proofpoint\\n4.0',\n",
       " 'PubMatic\\n4.2',\n",
       " 'PubNub\\n4.0',\n",
       " 'Punchh\\n4.1',\n",
       " 'Pyramid Consulting, Inc\\n3.9',\n",
       " 'Qualys\\n3.6',\n",
       " 'Quantifind\\n4.2',\n",
       " 'RAPP\\n3.6',\n",
       " 'RAPP\\n3.7',\n",
       " 'REDICA Systems',\n",
       " 'RELX Group\\n3.1',\n",
       " 'Randstad General Staffing\\n3.9',\n",
       " 'Raybeam, Inc.\\n4.8',\n",
       " 'Riiid Labs\\n5.0',\n",
       " 'Rivian Automotive\\n4.3',\n",
       " 'Robby Technologies\\n4.0',\n",
       " 'Robert Half Corporate Internal\\n3.7',\n",
       " 'Robinhood\\n4.3',\n",
       " 'Robinhood\\n4.4',\n",
       " 'Roblox\\n4.4',\n",
       " 'Roblox\\n4.5',\n",
       " 'Roche\\n4.3',\n",
       " 'Ryder System\\n3.5',\n",
       " 'SACC INC\\n3.5',\n",
       " 'SADA\\n4.8',\n",
       " 'SKT Lab',\n",
       " 'SRA OSS\\n3.5',\n",
       " 'SRG\\n3.9',\n",
       " 'STAR Labs\\n3.5',\n",
       " 'SafeTraces, Inc.',\n",
       " 'Salesforce\\n4.4',\n",
       " 'Salesforce\\n4.5',\n",
       " 'Samsung Electronics America\\n3.5',\n",
       " 'Samsung SDS America\\n3.5',\n",
       " 'ScaleMonk',\n",
       " 'Seasoned, LLC\\n4.7',\n",
       " 'Second Genome\\n4.7',\n",
       " 'Secureworks\\n4.2',\n",
       " 'Seer\\n5.0',\n",
       " 'Sensor Tower\\n4.0',\n",
       " 'Sentient Energy\\n3.1',\n",
       " 'Sentient Energy\\n3.9',\n",
       " 'Shoe Palace Corporation\\n3.3',\n",
       " 'Shoe Palace Corporation\\n3.5',\n",
       " 'Sightline Payments\\n4.4',\n",
       " 'Sigma Inc\\n4.1',\n",
       " 'Sigmaways, Inc.\\n3.9',\n",
       " 'Sikka Software\\n3.8',\n",
       " 'SiriusXM\\n3.7',\n",
       " 'SiriusXM\\n3.8',\n",
       " 'Sisu',\n",
       " 'Sitecore\\n3.6',\n",
       " 'Sitecore\\n3.7',\n",
       " 'Skan\\n4.8',\n",
       " 'Slack\\n4.7',\n",
       " 'Slalom LLC.\\n4.3',\n",
       " 'Snapchat\\n4.1',\n",
       " 'Snapchat\\n4.2',\n",
       " 'SoFi\\n3.6',\n",
       " 'Sony Corporation of America\\n3.1',\n",
       " 'Sony Interactive Entertainment PlayStation\\n4.0',\n",
       " 'Sony PlayStation\\n4.0',\n",
       " 'SpanIdea\\n3.8',\n",
       " 'Spartan Technologies\\n3.0',\n",
       " 'Specific Diagnostics\\n3.5',\n",
       " 'Spin\\n3.5',\n",
       " 'Spin\\n4.3',\n",
       " 'Splunk\\n4.1',\n",
       " 'Splunk\\n4.2',\n",
       " 'Spring Discovery\\n4.0',\n",
       " 'Squadex\\n3.0',\n",
       " 'Square\\n4.3',\n",
       " 'Stanford University\\n4.2',\n",
       " 'Stealth Mode Startup\\n4.0',\n",
       " 'Stealth Mode Startup\\n4.1',\n",
       " 'Stealth Mode Startup Company',\n",
       " 'Step\\n2.6',\n",
       " 'Step Mobile\\n4.2',\n",
       " 'Stitch Fix\\n3.7',\n",
       " 'Store No. 8\\n4.4',\n",
       " 'Strategi.biz',\n",
       " 'StubHub\\n3.4',\n",
       " 'Sunshine',\n",
       " 'SupportLogic\\n5.0',\n",
       " 'Surya Systems\\n4.0',\n",
       " 'Susquehanna International Group\\n4.0',\n",
       " 'Sutro Biopharma\\n4.1',\n",
       " 'SymphonyAI\\n4.3',\n",
       " 'Synapse Financial Technologies Inc\\n3.8',\n",
       " 'Synopsys\\n4.1',\n",
       " 'TCL Research America\\n4.0',\n",
       " 'TEKPLAY SYSTEMS\\n3.6',\n",
       " 'TEKsystems\\n3.8',\n",
       " 'Tailored Management\\n4.4',\n",
       " 'Tapjoy\\n3.6',\n",
       " 'TargetCW\\n4.4',\n",
       " 'TaskRabbit\\n4.0',\n",
       " 'TechTrueUP\\n3.6',\n",
       " 'Technosoft Global',\n",
       " 'Teknatio\\n5.0',\n",
       " 'Tekrek Solutions',\n",
       " 'Teladoc Health\\n4.0',\n",
       " 'Tempus\\n3.3',\n",
       " 'Tempus Labs\\n3.2',\n",
       " 'Tesla\\n3.8',\n",
       " 'The Climate Corporation\\n3.5',\n",
       " 'The Climate Corporation\\n3.6',\n",
       " 'The Mom Project\\n4.8',\n",
       " 'The Rockridge Group\\n2.3',\n",
       " 'Thermo Fisher - America\\n3.8',\n",
       " 'Thermo Fisher Scientific\\n3.8',\n",
       " 'ThirdLove\\n3.6',\n",
       " 'Tiger Analytics\\n4.4',\n",
       " 'TikTok\\n3.8',\n",
       " 'Tinder\\n3.7',\n",
       " 'Tinder\\n4.0',\n",
       " 'Toyota Research Institute\\n3.8',\n",
       " 'TuSimple, Inc.\\n4.4',\n",
       " 'Tubi\\n4.2',\n",
       " 'Turo\\n4.5',\n",
       " 'Turo\\n4.6',\n",
       " 'Twilio\\n4.4',\n",
       " 'Twist Bioscience\\n4.2',\n",
       " 'Twitch\\n3.9',\n",
       " 'Twitch Interactive, Inc.\\n3.8',\n",
       " 'Twitter\\n4.2',\n",
       " 'UC Hastings\\n3.6',\n",
       " 'UC San Francisco Academic\\n4.1',\n",
       " 'UC San Francisco Academic\\n4.2',\n",
       " 'Uber\\n4.0',\n",
       " 'Udemy\\n4.7',\n",
       " 'Ultimate Staffing Services\\n3.3',\n",
       " 'UnifyID\\n4.2',\n",
       " 'Uniphore\\n4.4',\n",
       " 'Universities Space Research Association\\n3.6',\n",
       " 'University of California Berkeley\\n4.4',\n",
       " 'University of California San Francisco\\n4.1',\n",
       " 'University of California San Francisco\\n4.2',\n",
       " 'University of California Santa Cruz\\n4.1',\n",
       " 'University of California, Davis\\n4.4',\n",
       " 'Unlearn.AI\\n5.0',\n",
       " 'Unstoppable Domains\\n4.9',\n",
       " 'Upstart\\n4.4',\n",
       " 'Upstart Network, Inc.\\n4.4',\n",
       " 'Ursus\\n4.4',\n",
       " 'UserLeap\\n5.0',\n",
       " 'V-Soft Consulting Group, Inc.\\n3.7',\n",
       " 'VIZIO, Inc.\\n4.0',\n",
       " 'VIZIO, Inc.\\n4.1',\n",
       " 'VMware\\n4.3',\n",
       " 'Varo Bank\\n3.7',\n",
       " 'Varo Money\\n3.7',\n",
       " 'Vectra\\n2.8',\n",
       " 'Veeva Systems\\n4.1',\n",
       " 'Verana Health\\n3.8',\n",
       " 'Verb Surgical\\n3.5',\n",
       " 'Verkada\\n4.0',\n",
       " 'Viaduct\\n5.0',\n",
       " 'Visual Concepts\\n4.2',\n",
       " 'Vitria Technology\\n3.5',\n",
       " 'Vungle\\n4.0',\n",
       " 'Walmart\\n3.4',\n",
       " \"Warman O'Brien\",\n",
       " 'Waymo\\n4.0',\n",
       " 'Waymo\\n4.1',\n",
       " 'WeDriveU, Inc.\\n3.6',\n",
       " 'Wealthfront\\n4.5',\n",
       " 'Webhelp Americas\\n3.2',\n",
       " 'Wells Fargo\\n3.6',\n",
       " 'Wells Fargo Bank\\n3.6',\n",
       " 'WhoWhatWhy\\n2.3',\n",
       " 'Wider Circle\\n4.8',\n",
       " 'Wild Type\\n5.0',\n",
       " 'Williams Sonoma\\n3.1',\n",
       " 'Wing\\n2.1',\n",
       " 'Wish\\n3.3',\n",
       " 'Wish\\n3.4',\n",
       " 'Woodruff Sawyer\\n4.0',\n",
       " 'Workday\\n4.3',\n",
       " 'Workforce Logiq\\n4.1',\n",
       " 'Wyzant\\n4.1',\n",
       " 'Xerox\\n3.2',\n",
       " 'Xerox\\n3.3',\n",
       " 'Xoriant Corporation\\n4.0',\n",
       " 'Yardzen\\n5.0',\n",
       " 'ZS Associates\\n4.0',\n",
       " 'Zeta Global\\n3.7',\n",
       " 'Zipcar, Inc.\\n3.5',\n",
       " 'Zipline\\n4.3',\n",
       " 'Zscaler\\n4.7',\n",
       " 'Zymergen\\n3.2',\n",
       " 'a.k.a. Brands',\n",
       " 'centraprise',\n",
       " 'dunnhumby\\n4.1',\n",
       " 'eBay\\n4.0',\n",
       " 'eBay Inc.\\n4.0',\n",
       " 'eHealth\\n3.7',\n",
       " 'f5\\n4.1',\n",
       " 'freshworks\\n4.3',\n",
       " 'iFoodDecisionSciences',\n",
       " 'infolob\\n4.3',\n",
       " 'kover\\n4.0',\n",
       " 'test.ai',\n",
       " 'tunein\\n4.4'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(sorted(df_clean.company.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ❓ Questions: 3.\\\n",
    "  \\\n",
    "  Which companies do we need to aggregate names for?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate: Amazon, Parker Institute for/of Cancer Immunotherapy, Sony, Stealth Mode Startup, Twitch, Wells Fargo, Varo\\\n",
    "Remove: financial services company, information technology sector\n",
    "\n",
    "Let's build out the cleaner for both splitting the ratings and handling these aggregations and removes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "replace_company_names = {\"Amazon\": \"Amazon\", \n",
    "                         \"Parker Institute\": \"Parker Institute for Cancer Immunotherapy\", \n",
    "                         \"Sony\": \"Sony\", \n",
    "                         \"Stealth Mode\": \"Stealth Mode\", \n",
    "                         \"Twitch\": \"Twitch\", \n",
    "                         \"Wells Fargo\": \"Wells Fargo\", \n",
    "                         \"Varo\": \"Varo\"}\n",
    "\n",
    "remove_company_names = [\"financial services company\", \"information technology sector\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Split company and rating and change rating dtype.\n",
    "# 2. Aggregate similar company names. \n",
    "# 3. Remove non-company names. \n",
    "\n",
    "class CompanyRatingSplit(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, idx=1):\n",
    "        self.idx = idx\n",
    "        \n",
    "    def _split_rating(self, X):\n",
    "        X_copy = X.copy()\n",
    "        company_rating = (X_copy[\"company\"]\n",
    "                          .apply(lambda company: pd.Series(company.split(\"\\n\")))\n",
    "                          .set_axis([\"company\", \"rating\"], axis=1))\n",
    "        X_copy[\"company\"] = company_rating[\"company\"]\n",
    "        X_copy.insert(self.idx, \"rating\", company_rating[\"rating\"])\n",
    "        X_copy.rating = X_copy.rating.astype(\"float64\")\n",
    "        \n",
    "        return X_copy\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "        \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        X_copy = self._split_rating(X_copy)\n",
    "        \n",
    "        return X_copy\n",
    "    \n",
    "class AggregateCompanyNames(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, company_names):\n",
    "        self.company_names = company_names\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        # Ideally, you would have a learnable algorithm to find similar names.\n",
    "        # Or, you have some simple NLP method to match similar words together.\n",
    "        # But this gets finnicky when you encounter \"University of California, Berkeley\"\n",
    "        # and \"University of California, Davis\".\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        for k, v in self.company_names.items():\n",
    "            filter_ = X_copy[\"company\"].apply(lambda company: True if k.lower() in company.lower() else False)\n",
    "            X_copy.loc[filter_, \"company\"] = v\n",
    "            \n",
    "        return X_copy.reset_index(drop=True)\n",
    "    \n",
    "class RemoveNonCompanyNames(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, company_names):\n",
    "        self.company_names = company_names\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        for c in self.company_names:\n",
    "            X_copy = X_copy[X_copy.company != c]\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know from a previous observation that there are NaNs in the ratings. Let's first fill NaNs by the company rating (if the commpany has multiple joblistings and some of them have ratings). Then, we will fill by global average rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Fill rating by average company rating.\n",
    "# 2. Fill by average rating.\n",
    "    \n",
    "class FillRatingByCmpnyAvg(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def _fill_nans_with_cmpny_avg_rating(self, cmpny_set, df):\n",
    "        for company in cmpny_set:\n",
    "            ratings = df[df.company == company][\"rating\"]\n",
    "            ratings_nans = ratings[ratings.isna().values]\n",
    "            nan_idx = list(ratings_nans.index)\n",
    "            ratings_avg = round(ratings.mean(), 1)\n",
    "            for idx in nan_idx:\n",
    "                df.at[idx, \"rating\"] = ratings_avg\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        companies = X_copy[\"company\"].value_counts()\n",
    "        idx_of_missing_rates = [company for company in companies.index if \"\\n\" not in company]\n",
    "        avg_cmpny_ratings = X_copy.groupby(\"company\").mean()[\"rating\"]\n",
    "        no_rating = avg_cmpny_ratings.isna().values\n",
    "        no_rating_cmpny = avg_cmpny_ratings.index[no_rating]\n",
    "        \n",
    "        self.companies_with_ratings = set(idx_of_missing_rates).difference(set(no_rating_cmpny))\n",
    "        \n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        self._fill_nans_with_cmpny_avg_rating(self.companies_with_ratings, X_copy)\n",
    "        \n",
    "        return X_copy\n",
    "    \n",
    "class FillRatingByAvg(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        self.avg_rating = round(X_copy.rating.mean(), 1)\n",
    "        \n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy.rating = X_copy.rating.fillna(self.avg_rating)\n",
    "        \n",
    "        return X_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_company_pipeline = Pipeline([\n",
    "    (\"company_rating_split\", CompanyRatingSplit()),\n",
    "    (\"company_aggregate\", AggregateCompanyNames(replace_company_names)),\n",
    "    (\"company_remove\", RemoveNonCompanyNames(remove_company_names))\n",
    "])\n",
    "\n",
    "clean_rating_pipeline = Pipeline([\n",
    "    (\"rating_fill_by_cmpny_avg\", FillRatingByCmpnyAvg()),\n",
    "    (\"rating_fill_by_avg\", FillRatingByAvg())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = clean_company_pipeline.fit_transform(df_clean)\n",
    "df_clean = clean_rating_pipeline.fit_transform(df_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Facebook                           119\n",
       "Ursus                               51\n",
       "Uber                                43\n",
       "Salesforce                          42\n",
       "Amazon                              41\n",
       "                                  ... \n",
       "Lemonaid Health                      1\n",
       "Intelliswift Software Inc            1\n",
       "University of California, Davis      1\n",
       "Grid Dynamics                        1\n",
       "EPM Scientific                       1\n",
       "Name: company, Length: 601, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.company.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📑 conclusions_univariate: [0, \"company\"].\\\n",
    "  \\\n",
    "  There seems to be a lot of companies posting just\\\n",
    "  one joblisting making it difficult to use for machine learning.\\\n",
    "  I could try different methods of sampling, but I'll hold off on this for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "for company in list(df_clean[\"company\"].values):\n",
    "    if \"-1\" in company: print(\"Missing company.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📑 conclusions_univariate: [1, \"company\"].\\\n",
    "  \\\n",
    "  Company has no missing values.\\\n",
    "  They also have no outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Walmart</td>\n",
       "      <td>3.4</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>retail</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TikTok</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>501 to 1000 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$187K - $231K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Employer Provided Salary:$130K - $156K</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America</td>\n",
       "      <td>3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   company  rating       headquarters  \\\n",
       "0                  Walmart     3.4      Sunnyvale, CA   \n",
       "1                   TikTok     3.8  Mountain View, CA   \n",
       "2                   Indeed     4.3  San Francisco, CA   \n",
       "3                   Indeed     4.3  San Francisco, CA   \n",
       "4  Thermo Fisher - America     3.8  San Francisco, CA   \n",
       "\n",
       "                          salary estimate              job type  \\\n",
       "0                                      -1        Job Type : N/A   \n",
       "1                                      -1  Job Type : Full-time   \n",
       "2  Employer Provided Salary:$187K - $231K  Job Type : Full-time   \n",
       "3  Employer Provided Salary:$130K - $156K  Job Type : Full-time   \n",
       "4                                      -1        Job Type : N/A   \n",
       "\n",
       "                    size               type                     sector  \\\n",
       "0       10000+ employees   company - public                     retail   \n",
       "1  501 to 1000 employees  company - private     information technology   \n",
       "2       10000+ employees  company - private     information technology   \n",
       "3       10000+ employees  company - private     information technology   \n",
       "4       10000+ employees   company - public  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue  \n",
       "0        $10+ billion (usd)  \n",
       "1  unknown / non-applicable  \n",
       "2    $2 to $5 billion (usd)  \n",
       "3    $2 to $5 billion (usd)  \n",
       "4        $10+ billion (usd)  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. headquarters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0,\n",
       " array(['Sunnyvale, CA', 'Mountain View, CA', 'San Francisco, CA',\n",
       "        'United States', 'San Jose, CA', 'Livermore, CA', 'Palo Alto, CA',\n",
       "        'Santa Clara, CA', 'Oakland, CA', 'San Mateo, CA', 'Los Altos, CA',\n",
       "        'Foster City, CA', 'Morgan Hill, CA', 'Newark, CA',\n",
       "        'Menlo Park, CA', 'Senior Data Scientist', 'Staff Data Scientist',\n",
       "        'Redwood City, CA', 'Cupertino, CA', 'Burlingame, CA',\n",
       "        'Emeryville, CA', 'South San Francisco, CA', 'Pleasanton, CA',\n",
       "        'Los Gatos, CA', 'Half Moon Bay, CA', 'Sausalito, CA',\n",
       "        'San Carlos, CA', 'Brisbane, CA', 'Elk Grove, CA',\n",
       "        'Business Data Analyst', 'Pittsburg, CA', 'Sacramento, CA',\n",
       "        'Fremont, CA', 'Concord, CA', 'East Palo Alto, CA', 'Milpitas, CA',\n",
       "        'Patterson, CA', 'Berkeley, CA', 'San Bruno, CA', 'Hercules, CA',\n",
       "        'San Ramon, CA', 'Staff Business Data Analysis',\n",
       "        'Economist - Machine Learning Engineer',\n",
       "        'Manager 3 Data and Analytics', 'Principal Data Scientist',\n",
       "        'Alameda, CA', 'Scotts Valley, CA', 'Petaluma, CA',\n",
       "        'Greenbrae, CA', 'Rancho Cordova, CA', 'Saint Helena, CA',\n",
       "        'Oakdale, CA', 'Clearlake, CA', 'Walnut Creek, CA', 'Lodi, CA',\n",
       "        'Stanford, CA', 'Staff Data Scientist - Product Analytics',\n",
       "        'Staff Business Data Analyst', 'Senior Technical Data Analyst',\n",
       "        'Tracy, CA', 'Hayward, CA', 'Staff Machine Learning Engineer',\n",
       "        'Group Manager - Data Science', 'Stockton, CA', 'Dublin, CA',\n",
       "        'Bellevue, CA', 'Richmond, CA', 'Monterey, CA',\n",
       "        'Staff Business Data Analyst, Ecosystem Fraud Prevention',\n",
       "        'Martinez, CA', 'Davis, CA', 'Senior Business Data Analyst',\n",
       "        'West Sacramento, CA', 'Novato, CA'], dtype=object))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.headquarters.isna().sum(), df_clean.headquarters.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📑 conclusions_univariate: [0, \"headquarters\"].\\\n",
    "  \\\n",
    "  There are no NaNs and no -1s, however, though the headquarters unique values\\\n",
    "  are mostly fine, there seem to be a lot of mix in with what seems to be\\\n",
    "  \"job title\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "San Francisco, CA               986\n",
       "Menlo Park, CA                  173\n",
       "Mountain View, CA               139\n",
       "San Jose, CA                    127\n",
       "Palo Alto, CA                   116\n",
       "                               ... \n",
       "Staff Business Data Analyst       1\n",
       "Tracy, CA                         1\n",
       "West Sacramento, CA               1\n",
       "Pittsburg, CA                     1\n",
       "Senior Business Data Analyst      1\n",
       "Name: headquarters, Length: 74, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "headquarter_value_counts = df_clean.headquarters.value_counts()\n",
    "headquarter_value_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"headquarters\"].\\\n",
    "  \\\n",
    "  We can omit the NaNs for where job titles are since there aren't that many."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ❓ Questions: 4.\\\n",
    "  \\\n",
    "  Can we extract all the non-headquarter unique entries?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['United States', 'Staff Machine Learning Engineer',\n",
       "       'Business Data Analyst', 'Senior Technical Data Analyst',\n",
       "       'Economist - Machine Learning Engineer', 'Principal Data Scientist',\n",
       "       'Senior Data Scientist', 'Staff Business Data Analysis',\n",
       "       'Staff Business Data Analyst, Ecosystem Fraud Prevention',\n",
       "       'Manager 3 Data and Analytics', 'Group Manager - Data Science',\n",
       "       'Staff Data Scientist', 'Staff Data Scientist - Product Analytics',\n",
       "       'Staff Business Data Analyst', 'Senior Business Data Analyst'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Since, all actual headquarter locations are located in California, \n",
    "# I will parse the indices and simply check which ones don't have CA.\n",
    "\n",
    "no_ca = headquarter_value_counts.index[headquarter_value_counts.index.map(lambda hq: \"CA\" not in hq)]\n",
    "no_ca"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [1, \"headquarters\"].\\\n",
    "  \\\n",
    "  United States is a location and it is one unique value in the \"headquarters\" column.\\\n",
    "  However, since CA is in the US, having the location of US doesn't help much if there aren't\\\n",
    "  joblistings also coming in internationally. We will remove US alongside all the \\\n",
    "  other job titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mitre Corporation</td>\n",
       "      <td>3.8</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>nonprofit organization</td>\n",
       "      <td>government</td>\n",
       "      <td>$1 to $2 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               company  rating   headquarters salary estimate  \\\n",
       "9    Mitre Corporation     3.8  United States              -1   \n",
       "43               Apple     4.3  United States              -1   \n",
       "49               Apple     4.3  United States              -1   \n",
       "55               Apple     4.3  United States              -1   \n",
       "64               Apple     4.3  United States              -1   \n",
       "70               Apple     4.3  United States              -1   \n",
       "94               Apple     4.3  United States              -1   \n",
       "96               Apple     4.3  United States              -1   \n",
       "105              Apple     4.3  United States              -1   \n",
       "\n",
       "                 job type                     size                    type  \\\n",
       "9    Job Type : Full-time  5001 to 10000 employees  nonprofit organization   \n",
       "43         Job Type : N/A         10000+ employees        company - public   \n",
       "49         Job Type : N/A         10000+ employees        company - public   \n",
       "55   Job Type : Full-time         10000+ employees        company - public   \n",
       "64   Job Type : Full-time         10000+ employees        company - public   \n",
       "70         Job Type : N/A         10000+ employees        company - public   \n",
       "94   Job Type : Full-time         10000+ employees        company - public   \n",
       "96   Job Type : Full-time         10000+ employees        company - public   \n",
       "105        Job Type : N/A         10000+ employees        company - public   \n",
       "\n",
       "                     sector                 revenue  \n",
       "9                government  $1 to $2 billion (usd)  \n",
       "43   information technology      $10+ billion (usd)  \n",
       "49   information technology      $10+ billion (usd)  \n",
       "55   information technology      $10+ billion (usd)  \n",
       "64   information technology      $10+ billion (usd)  \n",
       "70   information technology      $10+ billion (usd)  \n",
       "94   information technology      $10+ billion (usd)  \n",
       "96   information technology      $10+ billion (usd)  \n",
       "105  information technology      $10+ billion (usd)  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[df_clean.headquarters == \"United States\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [2, \"headquarters\"].\\\n",
    "  \\\n",
    "  Most jobs have a correct headquarters location. For those that don't, we will simply remove those instances from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mitre Corporation</td>\n",
       "      <td>3.8</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>nonprofit organization</td>\n",
       "      <td>government</td>\n",
       "      <td>$1 to $2 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>Apple</td>\n",
       "      <td>4.3</td>\n",
       "      <td>United States</td>\n",
       "      <td>-1</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>439</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analysis</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>734</th>\n",
       "      <td>Instacart - Corporate</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Economist - Machine Learning Engineer</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>1001 to 5000 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>741</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Manager 3 Data and Analytics</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>773</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Principal Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>783</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analysis</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>983</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Manager 3 Data and Analytics</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>988</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Data Scientist - Product Analytics</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1016</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1025</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Principal Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1075</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Technical Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1078</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Technical Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1133</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Technical Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1215</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Machine Learning Engineer</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1238</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Group Manager - Data Science</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1328</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Group Manager - Data Science</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analyst, Ecosystem Fraud P...</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1540</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analyst, Ecosystem Fraud P...</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1643</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Business Data Analysis</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1648</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1652</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1656</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Senior Business Data Analyst</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1779</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Principal Data Scientist</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>Instacart - Corporate</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Economist - Machine Learning Engineer</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>1001 to 5000 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Machine Learning Engineer</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Machine Learning Engineer</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2088</th>\n",
       "      <td>Instacart - Corporate</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Economist - Machine Learning Engineer</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>1001 to 5000 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2178</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Machine Learning Engineer</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2290</th>\n",
       "      <td>Intuit - Data</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Staff Machine Learning Engineer</td>\n",
       "      <td>Mountain View, CA</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>5001 to 10000 employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    company  rating  \\\n",
       "9         Mitre Corporation     3.8   \n",
       "43                    Apple     4.3   \n",
       "49                    Apple     4.3   \n",
       "55                    Apple     4.3   \n",
       "64                    Apple     4.3   \n",
       "70                    Apple     4.3   \n",
       "94                    Apple     4.3   \n",
       "96                    Apple     4.3   \n",
       "105                   Apple     4.3   \n",
       "142           Intuit - Data     4.5   \n",
       "143           Intuit - Data     4.5   \n",
       "275           Intuit - Data     4.5   \n",
       "439           Intuit - Data     4.5   \n",
       "578           Intuit - Data     4.5   \n",
       "734   Instacart - Corporate     3.8   \n",
       "741           Intuit - Data     4.5   \n",
       "773           Intuit - Data     4.5   \n",
       "783           Intuit - Data     4.5   \n",
       "977           Intuit - Data     4.5   \n",
       "983           Intuit - Data     4.5   \n",
       "988           Intuit - Data     4.5   \n",
       "999           Intuit - Data     4.5   \n",
       "1000          Intuit - Data     4.5   \n",
       "1006          Intuit - Data     4.5   \n",
       "1016          Intuit - Data     4.5   \n",
       "1025          Intuit - Data     4.5   \n",
       "1075          Intuit - Data     4.5   \n",
       "1078          Intuit - Data     4.5   \n",
       "1133          Intuit - Data     4.5   \n",
       "1215          Intuit - Data     4.5   \n",
       "1238          Intuit - Data     4.5   \n",
       "1328          Intuit - Data     4.5   \n",
       "1457          Intuit - Data     4.5   \n",
       "1540          Intuit - Data     4.5   \n",
       "1643          Intuit - Data     4.5   \n",
       "1648          Intuit - Data     4.5   \n",
       "1652          Intuit - Data     4.5   \n",
       "1656          Intuit - Data     4.5   \n",
       "1779          Intuit - Data     4.5   \n",
       "1971  Instacart - Corporate     3.8   \n",
       "2005          Intuit - Data     4.5   \n",
       "2006          Intuit - Data     4.5   \n",
       "2088  Instacart - Corporate     3.8   \n",
       "2178          Intuit - Data     4.5   \n",
       "2290          Intuit - Data     4.5   \n",
       "\n",
       "                                           headquarters    salary estimate  \\\n",
       "9                                         United States                 -1   \n",
       "43                                        United States                 -1   \n",
       "49                                        United States                 -1   \n",
       "55                                        United States                 -1   \n",
       "64                                        United States                 -1   \n",
       "70                                        United States                 -1   \n",
       "94                                        United States                 -1   \n",
       "96                                        United States                 -1   \n",
       "105                                       United States                 -1   \n",
       "142                               Senior Data Scientist  Mountain View, CA   \n",
       "143                                Staff Data Scientist  Mountain View, CA   \n",
       "275                               Business Data Analyst  Mountain View, CA   \n",
       "439                               Business Data Analyst  Mountain View, CA   \n",
       "578                        Staff Business Data Analysis  Mountain View, CA   \n",
       "734               Economist - Machine Learning Engineer  San Francisco, CA   \n",
       "741                        Manager 3 Data and Analytics  Mountain View, CA   \n",
       "773                            Principal Data Scientist  Mountain View, CA   \n",
       "783                               Senior Data Scientist  Mountain View, CA   \n",
       "977                        Staff Business Data Analysis  Mountain View, CA   \n",
       "983                        Manager 3 Data and Analytics  Mountain View, CA   \n",
       "988                               Business Data Analyst  Mountain View, CA   \n",
       "999            Staff Data Scientist - Product Analytics  Mountain View, CA   \n",
       "1000                        Staff Business Data Analyst  Mountain View, CA   \n",
       "1006                               Staff Data Scientist  Mountain View, CA   \n",
       "1016                              Senior Data Scientist  Mountain View, CA   \n",
       "1025                           Principal Data Scientist  Mountain View, CA   \n",
       "1075                      Senior Technical Data Analyst  Mountain View, CA   \n",
       "1078                      Senior Technical Data Analyst  Mountain View, CA   \n",
       "1133                      Senior Technical Data Analyst  Mountain View, CA   \n",
       "1215                    Staff Machine Learning Engineer  Mountain View, CA   \n",
       "1238                       Group Manager - Data Science  Mountain View, CA   \n",
       "1328                       Group Manager - Data Science  Mountain View, CA   \n",
       "1457  Staff Business Data Analyst, Ecosystem Fraud P...  Mountain View, CA   \n",
       "1540  Staff Business Data Analyst, Ecosystem Fraud P...  Mountain View, CA   \n",
       "1643                       Staff Business Data Analysis  Mountain View, CA   \n",
       "1648                              Business Data Analyst  Mountain View, CA   \n",
       "1652                              Business Data Analyst  Mountain View, CA   \n",
       "1656                       Senior Business Data Analyst  Mountain View, CA   \n",
       "1779                           Principal Data Scientist  Mountain View, CA   \n",
       "1971              Economist - Machine Learning Engineer  San Francisco, CA   \n",
       "2005                    Staff Machine Learning Engineer  Mountain View, CA   \n",
       "2006                    Staff Machine Learning Engineer  Mountain View, CA   \n",
       "2088              Economist - Machine Learning Engineer  San Francisco, CA   \n",
       "2178                    Staff Machine Learning Engineer  Mountain View, CA   \n",
       "2290                    Staff Machine Learning Engineer  Mountain View, CA   \n",
       "\n",
       "                  job type                     size                    type  \\\n",
       "9     Job Type : Full-time  5001 to 10000 employees  nonprofit organization   \n",
       "43          Job Type : N/A         10000+ employees        company - public   \n",
       "49          Job Type : N/A         10000+ employees        company - public   \n",
       "55    Job Type : Full-time         10000+ employees        company - public   \n",
       "64    Job Type : Full-time         10000+ employees        company - public   \n",
       "70          Job Type : N/A         10000+ employees        company - public   \n",
       "94    Job Type : Full-time         10000+ employees        company - public   \n",
       "96    Job Type : Full-time         10000+ employees        company - public   \n",
       "105         Job Type : N/A         10000+ employees        company - public   \n",
       "142   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "143   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "275   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "439   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "578   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "734   Job Type : Full-time   1001 to 5000 employees       company - private   \n",
       "741   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "773   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "783   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "977   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "983   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "988   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "999   Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1000  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1006  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1016  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1025  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1075  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1078  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1133  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1215  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1238  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1328  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1457  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1540  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1643  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1648  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1652  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1656  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1779  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "1971  Job Type : Full-time   1001 to 5000 employees       company - private   \n",
       "2005  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "2006  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "2088  Job Type : Full-time   1001 to 5000 employees       company - private   \n",
       "2178  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "2290  Job Type : Full-time  5001 to 10000 employees        company - public   \n",
       "\n",
       "                      sector                   revenue  \n",
       "9                 government    $1 to $2 billion (usd)  \n",
       "43    information technology        $10+ billion (usd)  \n",
       "49    information technology        $10+ billion (usd)  \n",
       "55    information technology        $10+ billion (usd)  \n",
       "64    information technology        $10+ billion (usd)  \n",
       "70    information technology        $10+ billion (usd)  \n",
       "94    information technology        $10+ billion (usd)  \n",
       "96    information technology        $10+ billion (usd)  \n",
       "105   information technology        $10+ billion (usd)  \n",
       "142   information technology    $2 to $5 billion (usd)  \n",
       "143   information technology    $2 to $5 billion (usd)  \n",
       "275   information technology    $2 to $5 billion (usd)  \n",
       "439   information technology    $2 to $5 billion (usd)  \n",
       "578   information technology    $2 to $5 billion (usd)  \n",
       "734   information technology  unknown / non-applicable  \n",
       "741   information technology    $2 to $5 billion (usd)  \n",
       "773   information technology    $2 to $5 billion (usd)  \n",
       "783   information technology    $2 to $5 billion (usd)  \n",
       "977   information technology    $2 to $5 billion (usd)  \n",
       "983   information technology    $2 to $5 billion (usd)  \n",
       "988   information technology    $2 to $5 billion (usd)  \n",
       "999   information technology    $2 to $5 billion (usd)  \n",
       "1000  information technology    $2 to $5 billion (usd)  \n",
       "1006  information technology    $2 to $5 billion (usd)  \n",
       "1016  information technology    $2 to $5 billion (usd)  \n",
       "1025  information technology    $2 to $5 billion (usd)  \n",
       "1075  information technology    $2 to $5 billion (usd)  \n",
       "1078  information technology    $2 to $5 billion (usd)  \n",
       "1133  information technology    $2 to $5 billion (usd)  \n",
       "1215  information technology    $2 to $5 billion (usd)  \n",
       "1238  information technology    $2 to $5 billion (usd)  \n",
       "1328  information technology    $2 to $5 billion (usd)  \n",
       "1457  information technology    $2 to $5 billion (usd)  \n",
       "1540  information technology    $2 to $5 billion (usd)  \n",
       "1643  information technology    $2 to $5 billion (usd)  \n",
       "1648  information technology    $2 to $5 billion (usd)  \n",
       "1652  information technology    $2 to $5 billion (usd)  \n",
       "1656  information technology    $2 to $5 billion (usd)  \n",
       "1779  information technology    $2 to $5 billion (usd)  \n",
       "1971  information technology  unknown / non-applicable  \n",
       "2005  information technology    $2 to $5 billion (usd)  \n",
       "2006  information technology    $2 to $5 billion (usd)  \n",
       "2088  information technology  unknown / non-applicable  \n",
       "2178  information technology    $2 to $5 billion (usd)  \n",
       "2290  information technology    $2 to $5 billion (usd)  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_extraneous = [True if \"CA\" not in hq else False for hq in df_clean.headquarters]\n",
    "df_clean[is_extraneous]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 misc_obsvtn: [1,].\\\n",
    "  \\\n",
    "  I was originally going to drop whatever rows have \"headquarters\" unique values\\\n",
    "  corresponding to job_titles however I realized that they\\\n",
    "  actually included the headquarters, it was just wrongly placed.\\\n",
    "  However, since they are all missing salary estimates (and I cannot infer\\\n",
    "  this salary estimate from similar instances), I will simply drop them as they\\\n",
    "  don't pose a significant contribution to my dataset (as there are only 9 instances).\\\n",
    "  Edit: Not 9 anymore with the introduction of a new batch of data,\\\n",
    "  but still negligible compared to the size of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build our cleaner pipeline for headquarters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Remove extraneous HQs.\n",
    "\n",
    "class RemoveExtraneousHQ(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        filter_ = [False if \"CA\" not in hq else True for hq in X_copy.headquarters]\n",
    "        \n",
    "        return X_copy[filter_].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_hq_pipeline = Pipeline([\n",
    "    (\"remove_ext_hq\", RemoveExtraneousHQ())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = clean_hq_pipeline.fit_transform(df_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Sunnyvale, CA', 'Mountain View, CA', 'San Francisco, CA',\n",
       "       'San Jose, CA', 'Livermore, CA', 'Palo Alto, CA',\n",
       "       'Santa Clara, CA', 'Oakland, CA', 'San Mateo, CA', 'Los Altos, CA',\n",
       "       'Foster City, CA', 'Morgan Hill, CA', 'Newark, CA',\n",
       "       'Menlo Park, CA', 'Redwood City, CA', 'Cupertino, CA',\n",
       "       'Burlingame, CA', 'Emeryville, CA', 'South San Francisco, CA',\n",
       "       'Pleasanton, CA', 'Los Gatos, CA', 'Half Moon Bay, CA',\n",
       "       'Sausalito, CA', 'San Carlos, CA', 'Brisbane, CA', 'Elk Grove, CA',\n",
       "       'Pittsburg, CA', 'Sacramento, CA', 'Fremont, CA', 'Concord, CA',\n",
       "       'East Palo Alto, CA', 'Milpitas, CA', 'Patterson, CA',\n",
       "       'Berkeley, CA', 'San Bruno, CA', 'Hercules, CA', 'San Ramon, CA',\n",
       "       'Alameda, CA', 'Scotts Valley, CA', 'Petaluma, CA',\n",
       "       'Greenbrae, CA', 'Rancho Cordova, CA', 'Saint Helena, CA',\n",
       "       'Oakdale, CA', 'Clearlake, CA', 'Walnut Creek, CA', 'Lodi, CA',\n",
       "       'Stanford, CA', 'Tracy, CA', 'Hayward, CA', 'Stockton, CA',\n",
       "       'Dublin, CA', 'Bellevue, CA', 'Richmond, CA', 'Monterey, CA',\n",
       "       'Martinez, CA', 'Davis, CA', 'West Sacramento, CA', 'Novato, CA'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.headquarters.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3. salary estimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['-1', 'Employer Provided Salary:$187K - $231K',\n",
       "       'Employer Provided Salary:$130K - $156K',\n",
       "       'Employer Provided Salary:$190K',\n",
       "       'Employer Provided Salary:$120K - $160K',\n",
       "       'Employer Provided Salary:$115K', '$99K - $170K (Glassdoor est.)',\n",
       "       '$94K - $169K (Glassdoor est.)', '$108K - $197K (Glassdoor est.)',\n",
       "       '$91K - $151K (Glassdoor est.)', '$63K - $133K (Glassdoor est.)',\n",
       "       '$126K - $222K (Glassdoor est.)', '$75K - $162K (Glassdoor est.)',\n",
       "       '$85K - $177K (Glassdoor est.)', '$101K - $179K (Glassdoor est.)',\n",
       "       '$96K - $154K (Glassdoor est.)', '$107K - $186K (Glassdoor est.)',\n",
       "       '$100K - $170K (Glassdoor est.)', '$43K - $79K (Glassdoor est.)',\n",
       "       '$80K - $151K (Glassdoor est.)', '$94K - $130K (Glassdoor est.)',\n",
       "       '$127K - $201K (Glassdoor est.)', '$83K - $159K (Glassdoor est.)',\n",
       "       '$82K - $167K (Glassdoor est.)', '$76K - $156K (Glassdoor est.)',\n",
       "       '$79K - $161K (Glassdoor est.)', '$90K - $174K (Glassdoor est.)',\n",
       "       '$92K - $160K (Glassdoor est.)', '$88K - $172K (Glassdoor est.)',\n",
       "       '$110K - $216K (Glassdoor est.)', '$84K - $175K (Glassdoor est.)',\n",
       "       '$95K - $163K (Glassdoor est.)', '$93K - $170K (Glassdoor est.)',\n",
       "       '$77K - $173K (Glassdoor est.)', '$90K - $170K (Glassdoor est.)',\n",
       "       '$80K - $154K (Glassdoor est.)', '$77K - $148K (Glassdoor est.)',\n",
       "       '$84K - $154K (Glassdoor est.)', '$141K - $155K (Glassdoor est.)',\n",
       "       '$96K - $182K (Glassdoor est.)', '$113K - $210K (Glassdoor est.)',\n",
       "       '$111K - $185K (Glassdoor est.)', '$84K - $179K (Glassdoor est.)',\n",
       "       '$91K - $164K (Glassdoor est.)', '$86K - $156K (Glassdoor est.)',\n",
       "       '$82K - $164K (Glassdoor est.)', '$86K - $174K (Glassdoor est.)',\n",
       "       '$87K - $154K (Glassdoor est.)', '$121K - $195K (Glassdoor est.)',\n",
       "       '$86K - $165K (Glassdoor est.)', '$102K - $185K (Glassdoor est.)',\n",
       "       '$108K - $174K (Glassdoor est.)', '$92K - $172K (Glassdoor est.)',\n",
       "       '$81K - $165K (Glassdoor est.)', '$100K - $165K (Glassdoor est.)',\n",
       "       '$79K - $160K (Glassdoor est.)', '$78K - $152K (Glassdoor est.)',\n",
       "       '$89K - $175K (Glassdoor est.)', '$68K - $142K (Glassdoor est.)',\n",
       "       '$121K - $200K (Glassdoor est.)', '$81K - $152K (Glassdoor est.)',\n",
       "       '$109K - $164K (Glassdoor est.)', '$121K - $209K (Glassdoor est.)',\n",
       "       '$78K - $150K (Glassdoor est.)', '$98K - $139K (Glassdoor est.)',\n",
       "       '$90K - $171K (Glassdoor est.)', '$98K - $174K (Glassdoor est.)',\n",
       "       '$94K - $154K (Glassdoor est.)', '$95K - $153K (Glassdoor est.)',\n",
       "       '$93K - $167K (Glassdoor est.)', '$142K - $176K (Glassdoor est.)',\n",
       "       '$80K - $159K (Glassdoor est.)', '$125K - $145K (Glassdoor est.)',\n",
       "       '$91K - $183K (Glassdoor est.)', '$96K - $157K (Glassdoor est.)',\n",
       "       '$89K - $165K (Glassdoor est.)', '$97K - $181K (Glassdoor est.)',\n",
       "       '$90K - $131K (Glassdoor est.)', '$93K - $157K (Glassdoor est.)',\n",
       "       '$99K - $180K (Glassdoor est.)', '$94K - $162K (Glassdoor est.)',\n",
       "       '$118K - $206K (Glassdoor est.)', '$91K - $178K (Glassdoor est.)',\n",
       "       '$93K - $195K (Glassdoor est.)', '$119K - $203K (Glassdoor est.)',\n",
       "       '$92K - $179K (Glassdoor est.)', '$91K - $161K (Glassdoor est.)',\n",
       "       '$105K - $170K (Glassdoor est.)', '$95K - $176K (Glassdoor est.)',\n",
       "       '$111K - $238K (Glassdoor est.)', '$104K - $180K (Glassdoor est.)',\n",
       "       '$97K - $154K (Glassdoor est.)', '$113K - $176K (Glassdoor est.)',\n",
       "       '$110K - $175K (Glassdoor est.)', '$83K - $141K (Glassdoor est.)',\n",
       "       '$90K - $143K (Glassdoor est.)', '$76K - $144K (Glassdoor est.)',\n",
       "       '$113K - $153K (Glassdoor est.)', '$83K - $167K (Glassdoor est.)',\n",
       "       '$96K - $105K (Glassdoor est.)', '$112K - $186K (Glassdoor est.)',\n",
       "       '$100K - $151K (Glassdoor est.)', '$141K - $166K (Glassdoor est.)',\n",
       "       '$89K - $168K (Glassdoor est.)', '$103K - $164K (Glassdoor est.)',\n",
       "       '$94K - $168K (Glassdoor est.)', '$115K - $124K (Glassdoor est.)',\n",
       "       '$124K - $220K (Glassdoor est.)', '$110K - $185K (Glassdoor est.)',\n",
       "       '$97K - $156K (Glassdoor est.)', '$72K - $147K (Glassdoor est.)',\n",
       "       '$77K - $159K (Glassdoor est.)', '$103K - $186K (Glassdoor est.)',\n",
       "       '$99K - $171K (Glassdoor est.)', '$137K - $219K (Glassdoor est.)',\n",
       "       '$86K - $154K (Glassdoor est.)', '$90K - $168K (Glassdoor est.)',\n",
       "       '$105K - $165K (Glassdoor est.)', '$91K - $165K (Glassdoor est.)',\n",
       "       '$113K - $163K (Glassdoor est.)', '$96K - $187K (Glassdoor est.)',\n",
       "       '$97K - $180K (Glassdoor est.)', '$86K - $172K (Glassdoor est.)',\n",
       "       '$142K - $168K (Glassdoor est.)', '$88K - $174K (Glassdoor est.)',\n",
       "       '$84K - $146K (Glassdoor est.)', '$67K - $130K (Glassdoor est.)',\n",
       "       '$99K - $177K (Glassdoor est.)', '$91K - $179K (Glassdoor est.)',\n",
       "       '$85K - $157K (Glassdoor est.)', '$89K - $180K (Glassdoor est.)',\n",
       "       '$104K - $207K (Glassdoor est.)', '$92K - $147K (Glassdoor est.)',\n",
       "       '$104K - $188K (Glassdoor est.)', '$84K - $141K (Glassdoor est.)',\n",
       "       '$68K - $111K (Glassdoor est.)', '$72K - $135K (Glassdoor est.)',\n",
       "       '$84K - $149K (Glassdoor est.)', '$59K - $138K (Glassdoor est.)',\n",
       "       '$74K - $153K (Glassdoor est.)', '$87K - $158K (Glassdoor est.)',\n",
       "       '$102K - $237K (Glassdoor est.)', '$98K - $172K (Glassdoor est.)',\n",
       "       '$101K - $187K (Glassdoor est.)', '$143K - $225K (Glassdoor est.)',\n",
       "       '$85K - $154K (Glassdoor est.)', '$96K - $181K (Glassdoor est.)',\n",
       "       '$84K - $161K (Glassdoor est.)', '$151K - $181K (Glassdoor est.)',\n",
       "       '$100K - $196K (Glassdoor est.)', '$79K - $156K (Glassdoor est.)',\n",
       "       '$80K - $134K (Glassdoor est.)', '$88K - $156K (Glassdoor est.)',\n",
       "       '$77K - $132K (Glassdoor est.)', '$91K - $159K (Glassdoor est.)',\n",
       "       '$78K - $128K (Glassdoor est.)', '$80K - $152K (Glassdoor est.)',\n",
       "       '$87K - $142K (Glassdoor est.)', '$90K - $180K (Glassdoor est.)',\n",
       "       '$117K - $187K (Glassdoor est.)', '$88K - $149K (Glassdoor est.)',\n",
       "       '$121K - $211K (Glassdoor est.)', '$126K - $199K (Glassdoor est.)',\n",
       "       '$74K - $145K (Glassdoor est.)', '$94K - $178K (Glassdoor est.)',\n",
       "       '$118K - $188K (Glassdoor est.)', '$90K - $151K (Glassdoor est.)',\n",
       "       '$104K - $166K (Glassdoor est.)', '$111K - $204K (Glassdoor est.)',\n",
       "       '$108K - $182K (Glassdoor est.)', '$100K - $181K (Glassdoor est.)',\n",
       "       '$93K - $178K (Glassdoor est.)', '$94K - $183K (Glassdoor est.)',\n",
       "       '$119K - $188K (Glassdoor est.)', '$86K - $164K (Glassdoor est.)',\n",
       "       '$130K - $145K (Glassdoor est.)', '$77K - $141K (Glassdoor est.)',\n",
       "       '$91K - $173K (Glassdoor est.)', '$77K - $155K (Glassdoor est.)',\n",
       "       '$90K - $159K (Glassdoor est.)', '$82K - $138K (Glassdoor est.)',\n",
       "       '$95K - $164K (Glassdoor est.)', '$131K - $150K (Glassdoor est.)',\n",
       "       '$99K - $160K (Glassdoor est.)', '$112K - $203K (Glassdoor est.)',\n",
       "       '$98K - $178K (Glassdoor est.)', '$112K - $181K (Glassdoor est.)',\n",
       "       '$68K - $135K (Glassdoor est.)', '$68K - $140K (Glassdoor est.)',\n",
       "       '$111K - $177K (Glassdoor est.)', '$127K - $212K (Glassdoor est.)',\n",
       "       '$85K - $165K (Glassdoor est.)', '$94K - $188K (Glassdoor est.)',\n",
       "       '$83K - $152K (Glassdoor est.)', '$96K - $152K (Glassdoor est.)',\n",
       "       '$84K - $157K (Glassdoor est.)', '$112K - $199K (Glassdoor est.)',\n",
       "       '$61K - $107K (Glassdoor est.)', '$108K - $170K (Glassdoor est.)',\n",
       "       '$95K - $167K (Glassdoor est.)', '$118K - $214K (Glassdoor est.)',\n",
       "       '$120K - $190K (Glassdoor est.)', '$78K - $143K (Glassdoor est.)',\n",
       "       '$145K - $164K (Glassdoor est.)', '$102K - $205K (Glassdoor est.)',\n",
       "       '$82K - $152K (Glassdoor est.)', '$89K - $160K (Glassdoor est.)',\n",
       "       '$67K - $132K (Glassdoor est.)', '$88K - $180K (Glassdoor est.)',\n",
       "       '$74K - $137K (Glassdoor est.)', '$60K - $110K (Glassdoor est.)',\n",
       "       '$71K - $130K (Glassdoor est.)', '$78K - $158K (Glassdoor est.)',\n",
       "       '$125K - $204K (Glassdoor est.)', '$66K - $143K (Glassdoor est.)',\n",
       "       '$119K - $200K (Glassdoor est.)', '$106K - $169K (Glassdoor est.)',\n",
       "       '$118K - $184K (Glassdoor est.)', '$92K - $182K (Glassdoor est.)',\n",
       "       '$127K - $222K (Glassdoor est.)', '$118K - $193K (Glassdoor est.)',\n",
       "       '$60K - $127K (Glassdoor est.)', '$106K - $177K (Glassdoor est.)',\n",
       "       '$84K - $178K (Glassdoor est.)', '$157K - $249K (Glassdoor est.)',\n",
       "       '$106K - $195K (Glassdoor est.)', '$92K - $171K (Glassdoor est.)',\n",
       "       '$79K - $131K (Glassdoor est.)', '$94K - $184K (Glassdoor est.)',\n",
       "       '$88K - $176K (Glassdoor est.)', '$106K - $176K (Glassdoor est.)',\n",
       "       '$70K - $132K (Glassdoor est.)', '$178K - $200K (Glassdoor est.)',\n",
       "       '$182K - $212K (Glassdoor est.)', '$95K - $173K (Glassdoor est.)',\n",
       "       '$122K - $150K (Glassdoor est.)', '$100K - $193K (Glassdoor est.)',\n",
       "       '$107K - $198K (Glassdoor est.)', '$130K - $204K (Glassdoor est.)',\n",
       "       '$98K - $175K (Glassdoor est.)', '$116K - $190K (Glassdoor est.)',\n",
       "       '$73K - $130K (Glassdoor est.)', '$93K - $150K (Glassdoor est.)',\n",
       "       '$97K - $159K (Glassdoor est.)', '$64K - $162K (Glassdoor est.)',\n",
       "       '$96K - $140K (Glassdoor est.)', '$60K - $121K (Glassdoor est.)',\n",
       "       '$108K - $173K (Glassdoor est.)', '$97K - $208K (Glassdoor est.)',\n",
       "       '$90K - $146K (Glassdoor est.)', '$114K - $190K (Glassdoor est.)',\n",
       "       '$145K - $160K (Glassdoor est.)', '$120K - $208K (Glassdoor est.)',\n",
       "       '$141K - $176K (Glassdoor est.)', '$98K - $198K (Glassdoor est.)',\n",
       "       '$151K - $242K (Glassdoor est.)', '$88K - $150K (Glassdoor est.)',\n",
       "       '$105K - $182K (Glassdoor est.)', '$107K - $181K (Glassdoor est.)',\n",
       "       '$60K - $132K (Glassdoor est.)', '$105K - $188K (Glassdoor est.)',\n",
       "       '$108K - $117K (Glassdoor est.)', '$130K - $190K (Glassdoor est.)',\n",
       "       '$103K - $171K (Glassdoor est.)', '$89K - $157K (Glassdoor est.)',\n",
       "       '$122K - $191K (Glassdoor est.)', '$109K - $204K (Glassdoor est.)',\n",
       "       '$105K - $196K (Glassdoor est.)', '$89K - $169K (Glassdoor est.)',\n",
       "       '$88K - $181K (Glassdoor est.)', '$83K - $172K (Glassdoor est.)',\n",
       "       '$92K - $191K (Glassdoor est.)', '$98K - $185K (Glassdoor est.)',\n",
       "       '$98K - $186K (Glassdoor est.)', '$88K - $161K (Glassdoor est.)',\n",
       "       '$80K - $158K (Glassdoor est.)', '$107K - $203K (Glassdoor est.)',\n",
       "       '$102K - $188K (Glassdoor est.)', '$93K - $174K (Glassdoor est.)',\n",
       "       '$65K - $119K (Glassdoor est.)', '$99K - $184K (Glassdoor est.)',\n",
       "       '$109K - $203K (Glassdoor est.)', '$117K - $183K (Glassdoor est.)',\n",
       "       '$107K - $200K (Glassdoor est.)', '$126K - $208K (Glassdoor est.)',\n",
       "       '$96K - $193K (Glassdoor est.)', '$81K - $158K (Glassdoor est.)',\n",
       "       '$86K - $173K (Glassdoor est.)', '$113K - $193K (Glassdoor est.)',\n",
       "       '$100K - $166K (Glassdoor est.)', '$50K - $105K (Glassdoor est.)',\n",
       "       '$96K - $206K (Glassdoor est.)', '$146K - $242K (Glassdoor est.)',\n",
       "       '$104K - $189K (Glassdoor est.)', '$168K - $207K (Glassdoor est.)',\n",
       "       '$93K - $175K (Glassdoor est.)', '$78K - $144K (Glassdoor est.)',\n",
       "       '$150K - $165K (Glassdoor est.)', '$117K - $194K (Glassdoor est.)',\n",
       "       '$57K - $111K (Glassdoor est.)', '$73K - $133K (Glassdoor est.)',\n",
       "       '$55K - $119K (Glassdoor est.)', '$59K - $112K (Glassdoor est.)',\n",
       "       '$108K - $203K (Glassdoor est.)', '$111K - $190K (Glassdoor est.)',\n",
       "       '$95K - $189K (Glassdoor est.)', '$115K - $193K (Glassdoor est.)',\n",
       "       '$86K - $159K (Glassdoor est.)', '$85K - $153K (Glassdoor est.)',\n",
       "       '$92K - $189K (Glassdoor est.)', '$93K - $162K (Glassdoor est.)',\n",
       "       '$78K - $164K (Glassdoor est.)', '$134K - $218K (Glassdoor est.)',\n",
       "       '$105K - $202K (Glassdoor est.)', '$106K - $201K (Glassdoor est.)',\n",
       "       '$89K - $164K (Glassdoor est.)', '$84K - $136K (Glassdoor est.)',\n",
       "       '$98K - $192K (Glassdoor est.)', '$158K - $183K (Glassdoor est.)',\n",
       "       '$100K - $192K (Glassdoor est.)', '$99K - $189K (Glassdoor est.)',\n",
       "       '$82K - $157K (Glassdoor est.)', '$95K - $185K (Glassdoor est.)',\n",
       "       '$107K - $167K (Glassdoor est.)', '$71K - $155K (Glassdoor est.)',\n",
       "       '$104K - $174K (Glassdoor est.)', '$58K - $109K (Glassdoor est.)',\n",
       "       '$104K - $193K (Glassdoor est.)', '$95K - $200K (Glassdoor est.)',\n",
       "       '$165K - $177K (Glassdoor est.)', '$76K - $157K (Glassdoor est.)',\n",
       "       '$91K - $157K (Glassdoor est.)', '$53K - $106K (Glassdoor est.)',\n",
       "       '$88K - $169K (Glassdoor est.)', '$81K - $148K (Glassdoor est.)',\n",
       "       '$85K - $148K (Glassdoor est.)', '$82K - $174K (Glassdoor est.)',\n",
       "       '$107K - $194K (Glassdoor est.)', '$122K - $192K (Glassdoor est.)',\n",
       "       '$75K - $163K (Glassdoor est.)', '$113K - $189K (Glassdoor est.)',\n",
       "       '$55K - $97K (Glassdoor est.)', '$70K - $116K (Glassdoor est.)',\n",
       "       '$70K - $137K (Glassdoor est.)', '$96K - $180K (Glassdoor est.)',\n",
       "       '$127K - $247K (Glassdoor est.)', '$119K - $205K (Glassdoor est.)',\n",
       "       '$117K - $202K (Glassdoor est.)', '$105K - $204K (Glassdoor est.)',\n",
       "       '$100K - $185K (Glassdoor est.)', '$108K - $178K (Glassdoor est.)',\n",
       "       '$30K - $150K (Glassdoor est.)', '$97K - $169K (Glassdoor est.)',\n",
       "       '$74K - $127K (Glassdoor est.)', '$92K - $187K (Glassdoor est.)',\n",
       "       '$74K - $101K (Glassdoor est.)', '$99K - $175K (Glassdoor est.)',\n",
       "       '$116K - $192K (Glassdoor est.)', '$114K - $198K (Glassdoor est.)',\n",
       "       '$75K - $150K (Glassdoor est.)', '$150K - $246K (Glassdoor est.)',\n",
       "       '$145K - $170K (Glassdoor est.)', '$106K - $187K (Glassdoor est.)',\n",
       "       '$117K - $206K (Glassdoor est.)', '$148K - $165K (Glassdoor est.)',\n",
       "       '$95K - $203K (Glassdoor est.)', '$124K - $211K (Glassdoor est.)',\n",
       "       '$119K - $197K (Glassdoor est.)', '$94K - $200K (Glassdoor est.)',\n",
       "       '$95K - $182K (Glassdoor est.)', '$125K - $230K (Glassdoor est.)',\n",
       "       '$86K - $168K (Glassdoor est.)', '$96K - $170K (Glassdoor est.)',\n",
       "       '$80K - $142K (Glassdoor est.)', '$96K - $179K (Glassdoor est.)',\n",
       "       '$123K - $207K (Glassdoor est.)', '$89K - $136K (Glassdoor est.)',\n",
       "       '$87K - $160K (Glassdoor est.)', '$51K - $106K (Glassdoor est.)',\n",
       "       '$93K - $179K (Glassdoor est.)', '$129K - $203K (Glassdoor est.)',\n",
       "       '$126K - $158K (Glassdoor est.)', '$105K - $140K (Glassdoor est.)',\n",
       "       '$108K - $180K (Glassdoor est.)', '$121K - $160K (Glassdoor est.)',\n",
       "       '$93K - $173K (Glassdoor est.)', '$110K - $193K (Glassdoor est.)',\n",
       "       '$85K - $140K (Glassdoor est.)', '$85K - $147K (Glassdoor est.)',\n",
       "       '$74K - $150K (Glassdoor est.)', '$89K - $155K (Glassdoor est.)',\n",
       "       '$93K - $180K (Glassdoor est.)', '$94K - $180K (Glassdoor est.)',\n",
       "       '$103K - $201K (Glassdoor est.)', '$118K - $189K (Glassdoor est.)',\n",
       "       '$63K - $114K (Glassdoor est.)', '$96K - $185K (Glassdoor est.)',\n",
       "       '$96K - $172K (Glassdoor est.)', '$121K - $194K (Glassdoor est.)',\n",
       "       '$84K - $145K (Glassdoor est.)', '$125K - $198K (Glassdoor est.)',\n",
       "       '$102K - $166K (Glassdoor est.)', '$110K - $227K (Glassdoor est.)',\n",
       "       '$105K - $127K (Glassdoor est.)', '$97K - $195K (Glassdoor est.)',\n",
       "       '$83K - $154K (Glassdoor est.)', '$91K - $162K (Glassdoor est.)',\n",
       "       '$108K - $198K (Glassdoor est.)', '$102K - $182K (Glassdoor est.)',\n",
       "       '$79K - $136K (Glassdoor est.)', '$104K - $198K (Glassdoor est.)',\n",
       "       '$77K - $162K (Glassdoor est.)', '$93K - $164K (Glassdoor est.)',\n",
       "       '$162K - $162K (Glassdoor est.)', '$114K - $211K (Glassdoor est.)',\n",
       "       '$105K - $201K (Glassdoor est.)', '$97K - $183K (Glassdoor est.)',\n",
       "       '$96K - $160K (Glassdoor est.)', '$70K - $130K (Glassdoor est.)',\n",
       "       '$56K - $113K (Glassdoor est.)', '$139K - $171K (Glassdoor est.)',\n",
       "       '$86K - $161K (Glassdoor est.)', '$106K - $202K (Glassdoor est.)',\n",
       "       '$95K - $175K (Glassdoor est.)', '$106K - $205K (Glassdoor est.)',\n",
       "       '$103K - $193K (Glassdoor est.)', '$81K - $146K (Glassdoor est.)',\n",
       "       '$63K - $120K (Glassdoor est.)', '$101K - $185K (Glassdoor est.)',\n",
       "       '$85K - $193K (Glassdoor est.)', '$52K - $110K (Glassdoor est.)',\n",
       "       '$76K - $142K (Glassdoor est.)', '$89K - $207K (Glassdoor est.)',\n",
       "       '$82K - $154K (Glassdoor est.)', '$115K - $215K (Glassdoor est.)',\n",
       "       '$115K - $173K (Glassdoor est.)', '$59K - $124K (Glassdoor est.)',\n",
       "       '$80K - $137K (Glassdoor est.)', '$73K - $145K (Glassdoor est.)',\n",
       "       '$61K - $111K (Glassdoor est.)', '$92K - $180K (Glassdoor est.)',\n",
       "       '$54K - $108K (Glassdoor est.)', '$59K - $109K (Glassdoor est.)',\n",
       "       '$58K - $114K (Glassdoor est.)', '$94K - $195K (Glassdoor est.)',\n",
       "       '$106K - $186K (Glassdoor est.)', '$73K - $176K (Glassdoor est.)',\n",
       "       '$39K - $87K (Glassdoor est.)', '$57K - $118K (Glassdoor est.)',\n",
       "       '$53K - $117K (Glassdoor est.)',\n",
       "       '$19 - $30 Per Hour(Glassdoor est.)',\n",
       "       '$51K - $108K (Glassdoor est.)',\n",
       "       '$21 - $34 Per Hour(Glassdoor est.)',\n",
       "       '$98K - $217K (Glassdoor est.)', '$70K - $194K (Glassdoor est.)',\n",
       "       '$81K - $164K (Glassdoor est.)', '$83K - $148K (Glassdoor est.)',\n",
       "       '$95K - $199K (Glassdoor est.)', '$102K - $203K (Glassdoor est.)',\n",
       "       '$82K - $156K (Glassdoor est.)', '$125K - $261K (Glassdoor est.)',\n",
       "       '$103K - $207K (Glassdoor est.)', '$79K - $154K (Glassdoor est.)',\n",
       "       '$47K - $113K (Glassdoor est.)', '$79K - $146K (Glassdoor est.)',\n",
       "       '$82K - $171K (Glassdoor est.)', '$59K - $117K (Glassdoor est.)',\n",
       "       '$80K - $163K (Glassdoor est.)', '$70K - $149K (Glassdoor est.)',\n",
       "       '$85K - $164K (Glassdoor est.)', '$87K - $166K (Glassdoor est.)',\n",
       "       '$51K - $102K (Glassdoor est.)', '$96K - $183K (Glassdoor est.)',\n",
       "       '$82K - $169K (Glassdoor est.)', '$100K - $205K (Glassdoor est.)',\n",
       "       '$105K - $197K (Glassdoor est.)', '$77K - $138K (Glassdoor est.)',\n",
       "       '$99K - $191K (Glassdoor est.)', '$72K - $133K (Glassdoor est.)',\n",
       "       '$49K - $110K (Glassdoor est.)', '$80K - $149K (Glassdoor est.)',\n",
       "       '$92K - $216K (Glassdoor est.)', '$101K - $186K (Glassdoor est.)',\n",
       "       '$73K - $136K (Glassdoor est.)', '$66K - $121K (Glassdoor est.)',\n",
       "       '$82K - $158K (Glassdoor est.)', '$110K - $210K (Glassdoor est.)',\n",
       "       '$63K - $117K (Glassdoor est.)', '$116K - $223K (Glassdoor est.)',\n",
       "       '$115K - $205K (Glassdoor est.)', '$91K - $176K (Glassdoor est.)',\n",
       "       '$69K - $145K (Glassdoor est.)', '$58K - $112K (Glassdoor est.)',\n",
       "       '$96K - $188K (Glassdoor est.)', '$90K - $178K (Glassdoor est.)',\n",
       "       '$72K - $146K (Glassdoor est.)', '$105K - $191K (Glassdoor est.)',\n",
       "       '$108K - $193K (Glassdoor est.)', '$81K - $162K (Glassdoor est.)',\n",
       "       '$63K - $125K (Glassdoor est.)', '$149K - $179K (Glassdoor est.)',\n",
       "       '$64K - $132K (Glassdoor est.)', '$99K - $167K (Glassdoor est.)',\n",
       "       '$72K - $154K (Glassdoor est.)', '$95K - $180K (Glassdoor est.)',\n",
       "       '$103K - $182K (Glassdoor est.)', '$48K - $106K (Glassdoor est.)',\n",
       "       '$104K - $161K (Glassdoor est.)', '$109K - $187K (Glassdoor est.)',\n",
       "       '$106K - $196K (Glassdoor est.)', '$58K - $118K (Glassdoor est.)',\n",
       "       '$132K - $219K (Glassdoor est.)', '$89K - $176K (Glassdoor est.)',\n",
       "       '$97K - $167K (Glassdoor est.)', '$114K - $214K (Glassdoor est.)',\n",
       "       '$145K - $239K (Glassdoor est.)', '$93K - $171K (Glassdoor est.)',\n",
       "       '$94K - $152K (Glassdoor est.)', '$95K - $159K (Glassdoor est.)',\n",
       "       '$92K - $183K (Glassdoor est.)', '$88K - $167K (Glassdoor est.)',\n",
       "       '$82K - $134K (Glassdoor est.)', '$83K - $162K (Glassdoor est.)',\n",
       "       '$69K - $128K (Glassdoor est.)', '$100K - $157K (Glassdoor est.)',\n",
       "       '$76K - $162K (Glassdoor est.)', '$64K - $118K (Glassdoor est.)',\n",
       "       '$113K - $194K (Glassdoor est.)', '$75K - $142K (Glassdoor est.)',\n",
       "       '$122K - $220K (Glassdoor est.)', '$48K - $83K (Glassdoor est.)',\n",
       "       '$76K - $151K (Glassdoor est.)', '$100K - $209K (Glassdoor est.)',\n",
       "       '$108K - $190K (Glassdoor est.)', '$63K - $134K (Glassdoor est.)',\n",
       "       '$107K - $213K (Glassdoor est.)', '$81K - $171K (Glassdoor est.)',\n",
       "       '$55K - $99K (Glassdoor est.)', '$94K - $171K (Glassdoor est.)',\n",
       "       '$76K - $137K (Glassdoor est.)', '$109K - $218K (Glassdoor est.)',\n",
       "       '$99K - $207K (Glassdoor est.)', '$59K - $115K (Glassdoor est.)',\n",
       "       '$104K - $179K (Glassdoor est.)', '$76K - $153K (Glassdoor est.)',\n",
       "       '$73K - $141K (Glassdoor est.)', '$58K - $117K (Glassdoor est.)',\n",
       "       '$74K - $135K (Glassdoor est.)', '$79K - $147K (Glassdoor est.)',\n",
       "       '$57K - $105K (Glassdoor est.)', '$74K - $134K (Glassdoor est.)',\n",
       "       '$86K - $166K (Glassdoor est.)', '$64K - $140K (Glassdoor est.)',\n",
       "       '$80K - $145K (Glassdoor est.)', '$103K - $213K (Glassdoor est.)',\n",
       "       '$71K - $142K (Glassdoor est.)', '$68K - $134K (Glassdoor est.)',\n",
       "       '$78K - $141K (Glassdoor est.)', '$88K - $190K (Glassdoor est.)',\n",
       "       '$149K - $238K (Glassdoor est.)', '$62K - $126K (Glassdoor est.)',\n",
       "       '$68K - $133K (Glassdoor est.)', '$94K - $215K (Glassdoor est.)',\n",
       "       '$79K - $162K (Glassdoor est.)', '$64K - $133K (Glassdoor est.)',\n",
       "       '$88K - $166K (Glassdoor est.)', '$109K - $188K (Glassdoor est.)',\n",
       "       '$89K - $186K (Glassdoor est.)', '$118K - $199K (Glassdoor est.)',\n",
       "       '$91K - $195K (Glassdoor est.)', '$81K - $137K (Glassdoor est.)',\n",
       "       '$92K - $207K (Glassdoor est.)', '$59K - $114K (Glassdoor est.)',\n",
       "       '$141K - $257K (Glassdoor est.)', '$82K - $146K (Glassdoor est.)',\n",
       "       '$133K - $227K (Glassdoor est.)', '$67K - $140K (Glassdoor est.)',\n",
       "       '$83K - $193K (Glassdoor est.)', '$76K - $158K (Glassdoor est.)',\n",
       "       '$96K - $171K (Glassdoor est.)', '$102K - $173K (Glassdoor est.)',\n",
       "       '$79K - $139K (Glassdoor est.)', '$94K - $187K (Glassdoor est.)',\n",
       "       '$132K - $227K (Glassdoor est.)', '$50K - $112K (Glassdoor est.)',\n",
       "       '$48K - $90K (Glassdoor est.)', '$61K - $132K (Glassdoor est.)',\n",
       "       '$75K - $167K (Glassdoor est.)', '$55K - $114K (Glassdoor est.)',\n",
       "       '$126K - $225K (Glassdoor est.)', '$131K - $223K (Glassdoor est.)',\n",
       "       '$55K - $101K (Glassdoor est.)', '$95K - $190K (Glassdoor est.)',\n",
       "       '$46K - $103K (Glassdoor est.)', '$63K - $122K (Glassdoor est.)',\n",
       "       '$116K - $226K (Glassdoor est.)', '$98K - $181K (Glassdoor est.)',\n",
       "       '$80K - $157K (Glassdoor est.)', '$94K - $194K (Glassdoor est.)',\n",
       "       '$85K - $183K (Glassdoor est.)', '$81K - $159K (Glassdoor est.)',\n",
       "       '$91K - $190K (Glassdoor est.)', '$77K - $144K (Glassdoor est.)',\n",
       "       '$49K - $107K (Glassdoor est.)', '$45K - $82K (Glassdoor est.)',\n",
       "       '$67K - $125K (Glassdoor est.)', '$91K - $181K (Glassdoor est.)',\n",
       "       '$138K - $243K (Glassdoor est.)', '$61K - $129K (Glassdoor est.)',\n",
       "       '$100K - $194K (Glassdoor est.)', '$103K - $177K (Glassdoor est.)',\n",
       "       '$89K - $158K (Glassdoor est.)', '$116K - $213K (Glassdoor est.)',\n",
       "       '$44K - $83K (Glassdoor est.)', '$93K - $190K (Glassdoor est.)',\n",
       "       '$91K - $194K (Glassdoor est.)', '$96K - $226K (Glassdoor est.)',\n",
       "       '$55K - $153K (Glassdoor est.)', '$79K - $145K (Glassdoor est.)',\n",
       "       '$91K - $163K (Glassdoor est.)', '$100K - $188K (Glassdoor est.)',\n",
       "       '$102K - $191K (Glassdoor est.)', '$101K - $146K (Glassdoor est.)',\n",
       "       '$101K - $181K (Glassdoor est.)', '$107K - $216K (Glassdoor est.)',\n",
       "       '$75K - $151K (Glassdoor est.)', '$82K - $151K (Glassdoor est.)',\n",
       "       '$74K - $141K (Glassdoor est.)', '$118K - $186K (Glassdoor est.)',\n",
       "       '$81K - $161K (Glassdoor est.)', '$66K - $130K (Glassdoor est.)',\n",
       "       '$65K - $127K (Glassdoor est.)', '$82K - $172K (Glassdoor est.)',\n",
       "       '$76K - $154K (Glassdoor est.)', '$84K - $138K (Glassdoor est.)',\n",
       "       '$90K - $165K (Glassdoor est.)', '$80K - $161K (Glassdoor est.)',\n",
       "       '$94K - $179K (Glassdoor est.)', '$84K - $165K (Glassdoor est.)',\n",
       "       '$85K - $207K (Glassdoor est.)', '$105K - $186K (Glassdoor est.)',\n",
       "       '$56K - $93K (Glassdoor est.)', '$83K - $163K (Glassdoor est.)',\n",
       "       '$86K - $163K (Glassdoor est.)', '$133K - $240K (Glassdoor est.)',\n",
       "       '$78K - $184K (Glassdoor est.)', '$72K - $137K (Glassdoor est.)',\n",
       "       '$67K - $127K (Glassdoor est.)', '$100K - $174K (Glassdoor est.)',\n",
       "       '$91K - $191K (Glassdoor est.)', '$65K - $126K (Glassdoor est.)',\n",
       "       '$81K - $157K (Glassdoor est.)', '$75K - $140K (Glassdoor est.)',\n",
       "       '$67K - $139K (Glassdoor est.)', '$89K - $172K (Glassdoor est.)',\n",
       "       '$89K - $161K (Glassdoor est.)', '$101K - $193K (Glassdoor est.)',\n",
       "       '$125K - $227K (Glassdoor est.)', '$92K - $168K (Glassdoor est.)',\n",
       "       '$24K - $50K (Glassdoor est.)', '$75K - $149K (Glassdoor est.)',\n",
       "       '$74K - $155K (Glassdoor est.)', '$63K - $137K (Glassdoor est.)',\n",
       "       '$62K - $108K (Glassdoor est.)', '$70K - $135K (Glassdoor est.)',\n",
       "       '$48K - $91K (Glassdoor est.)', '$69K - $148K (Glassdoor est.)',\n",
       "       '$60K - $122K (Glassdoor est.)', '$76K - $149K (Glassdoor est.)',\n",
       "       '$96K - $192K (Glassdoor est.)', '$100K - $140K (Glassdoor est.)',\n",
       "       '$91K - $182K (Glassdoor est.)', '$78K - $195K (Glassdoor est.)',\n",
       "       '$40K - $77K (Glassdoor est.)', '$66K - $116K (Glassdoor est.)',\n",
       "       '$76K - $133K (Glassdoor est.)', '$101K - $203K (Glassdoor est.)',\n",
       "       '$114K - $215K (Glassdoor est.)', '$71K - $167K (Glassdoor est.)',\n",
       "       '$93K - $158K (Glassdoor est.)', '$45K - $90K (Glassdoor est.)',\n",
       "       '$104K - $203K (Glassdoor est.)', '$104K - $175K (Glassdoor est.)',\n",
       "       '$107K - $189K (Glassdoor est.)', '$66K - $124K (Glassdoor est.)',\n",
       "       '$113K - $207K (Glassdoor est.)', '$18K - $95K (Glassdoor est.)',\n",
       "       '$70K - $158K (Glassdoor est.)', '$120K - $125K (Glassdoor est.)',\n",
       "       '$96K - $199K (Glassdoor est.)', '$66K - $125K (Glassdoor est.)',\n",
       "       '$98K - $164K (Glassdoor est.)', '$102K - $192K (Glassdoor est.)',\n",
       "       '$70K - $141K (Glassdoor est.)', '$85K - $155K (Glassdoor est.)',\n",
       "       '$89K - $167K (Glassdoor est.)', '$67K - $129K (Glassdoor est.)',\n",
       "       '$58K - $110K (Glassdoor est.)', '$91K - $185K (Glassdoor est.)',\n",
       "       '$97K - $179K (Glassdoor est.)', '$101K - $198K (Glassdoor est.)',\n",
       "       '$96K - $194K (Glassdoor est.)', '$119K - $210K (Glassdoor est.)',\n",
       "       '$36 - $58 Per Hour(Glassdoor est.)',\n",
       "       '$120K - $220K (Glassdoor est.)', '$77K - $142K (Glassdoor est.)',\n",
       "       '$80K - $144K (Glassdoor est.)', '$103K - $203K (Glassdoor est.)',\n",
       "       '$123K - $211K (Glassdoor est.)', '$138K - $170K (Glassdoor est.)',\n",
       "       '$79K - $124K (Glassdoor est.)', '$92K - $192K (Glassdoor est.)',\n",
       "       '$131K - $136K (Glassdoor est.)', '$119K - $134K (Glassdoor est.)',\n",
       "       '$41K - $61K (Glassdoor est.)', '$54K - $129K (Glassdoor est.)',\n",
       "       '$90K - $185K (Glassdoor est.)', '$94K - $177K (Glassdoor est.)',\n",
       "       '$65K - $128K (Glassdoor est.)', '$107K - $201K (Glassdoor est.)',\n",
       "       '$64K - $130K (Glassdoor est.)', '$86K - $153K (Glassdoor est.)',\n",
       "       '$109K - $198K (Glassdoor est.)', '$90K - $184K (Glassdoor est.)',\n",
       "       '$84K - $168K (Glassdoor est.)', '$71K - $136K (Glassdoor est.)',\n",
       "       '$36K - $71K (Glassdoor est.)', '$62K - $113K (Glassdoor est.)',\n",
       "       '$65K - $124K (Glassdoor est.)', '$78K - $147K (Glassdoor est.)',\n",
       "       '$77K - $154K (Glassdoor est.)', '$90K - $169K (Glassdoor est.)',\n",
       "       '$76K - $171K (Glassdoor est.)', '$95K - $177K (Glassdoor est.)',\n",
       "       '$90K - $181K (Glassdoor est.)', '$111K - $187K (Glassdoor est.)',\n",
       "       '$122K - $193K (Glassdoor est.)', '$124K - $200K (Glassdoor est.)',\n",
       "       '$96K - $186K (Glassdoor est.)', '$92K - $148K (Glassdoor est.)',\n",
       "       '$91K - $186K (Glassdoor est.)', '$88K - $175K (Glassdoor est.)',\n",
       "       '$127K - $200K (Glassdoor est.)', '$62K - $121K (Glassdoor est.)',\n",
       "       '$75K - $144K (Glassdoor est.)', '$84K - $170K (Glassdoor est.)',\n",
       "       '$131K - $197K (Glassdoor est.)', '$77K - $151K (Glassdoor est.)',\n",
       "       '$86K - $177K (Glassdoor est.)', '$67K - $108K (Glassdoor est.)',\n",
       "       '$102K - $189K (Glassdoor est.)', '$102K - $178K (Glassdoor est.)',\n",
       "       '$80K - $147K (Glassdoor est.)', '$107K - $202K (Glassdoor est.)',\n",
       "       '$98K - $171K (Glassdoor est.)', '$99K - $186K (Glassdoor est.)',\n",
       "       '$99K - $188K (Glassdoor est.)', '$101K - $170K (Glassdoor est.)',\n",
       "       '$77K - $152K (Glassdoor est.)', '$83K - $164K (Glassdoor est.)',\n",
       "       '$85K - $166K (Glassdoor est.)', '$117K - $185K (Glassdoor est.)',\n",
       "       '$116K - $214K (Glassdoor est.)', '$87K - $167K (Glassdoor est.)',\n",
       "       '$112K - $172K (Glassdoor est.)', '$163K - $176K (Glassdoor est.)',\n",
       "       '$83K - $169K (Glassdoor est.)', '$97K - $173K (Glassdoor est.)',\n",
       "       '$116K - $211K (Glassdoor est.)', '$114K - $199K (Glassdoor est.)',\n",
       "       '$97K - $198K (Glassdoor est.)', '$118K - $185K (Glassdoor est.)',\n",
       "       '$89K - $148K (Glassdoor est.)', '$94K - $167K (Glassdoor est.)',\n",
       "       '$107K - $192K (Glassdoor est.)', '$96K - $173K (Glassdoor est.)',\n",
       "       '$91K - $167K (Glassdoor est.)', '$56K - $109K (Glassdoor est.)',\n",
       "       '$54K - $107K (Glassdoor est.)', '$74K - $147K (Glassdoor est.)',\n",
       "       '$116K - $212K (Glassdoor est.)', '$109K - $199K (Glassdoor est.)',\n",
       "       '$99K - $196K (Glassdoor est.)', '$85K - $169K (Glassdoor est.)',\n",
       "       '$82K - $173K (Glassdoor est.)', '$92K - $178K (Glassdoor est.)',\n",
       "       '$98K - $184K (Glassdoor est.)', '$122K - $144K (Glassdoor est.)',\n",
       "       '$97K - $177K (Glassdoor est.)', '$79K - $155K (Glassdoor est.)',\n",
       "       '$109K - $182K (Glassdoor est.)', '$55K - $93K (Glassdoor est.)',\n",
       "       '$94K - $160K (Glassdoor est.)', '$119K - $214K (Glassdoor est.)',\n",
       "       '$67K - $123K (Glassdoor est.)', '$92K - $196K (Glassdoor est.)',\n",
       "       '$89K - $159K (Glassdoor est.)', '$102K - $200K (Glassdoor est.)',\n",
       "       '$135K - $213K (Glassdoor est.)', '$69K - $171K (Glassdoor est.)',\n",
       "       '$107K - $191K (Glassdoor est.)', '$93K - $199K (Glassdoor est.)',\n",
       "       '$87K - $163K (Glassdoor est.)', '$142K - $223K (Glassdoor est.)',\n",
       "       '$85K - $170K (Glassdoor est.)', '$107K - $172K (Glassdoor est.)',\n",
       "       '$87K - $175K (Glassdoor est.)', '$114K - $216K (Glassdoor est.)',\n",
       "       '$125K - $226K (Glassdoor est.)', '$63K - $116K (Glassdoor est.)',\n",
       "       '$86K - $183K (Glassdoor est.)', '$107K - $204K (Glassdoor est.)',\n",
       "       '$70K - $128K (Glassdoor est.)', '$60K - $111K (Glassdoor est.)',\n",
       "       '$61K - $115K (Glassdoor est.)', '$58K - $124K (Glassdoor est.)',\n",
       "       '$74K - $138K (Glassdoor est.)', '$96K - $175K (Glassdoor est.)',\n",
       "       '$142K - $169K (Glassdoor est.)', '$78K - $173K (Glassdoor est.)',\n",
       "       '$83K - $183K (Glassdoor est.)', '$80K - $150K (Glassdoor est.)',\n",
       "       '$97K - $178K (Glassdoor est.)', '$108K - $210K (Glassdoor est.)',\n",
       "       '$71K - $137K (Glassdoor est.)', '$72K - $131K (Glassdoor est.)',\n",
       "       '$107K - $188K (Glassdoor est.)', '$106K - $197K (Glassdoor est.)',\n",
       "       '$100K - $187K (Glassdoor est.)', '$60K - $103K (Glassdoor est.)',\n",
       "       '$87K - $157K (Glassdoor est.)', '$170K - $182K (Glassdoor est.)',\n",
       "       '$88K - $127K (Glassdoor est.)', '$117K - $193K (Glassdoor est.)',\n",
       "       '$91K - $150K (Glassdoor est.)', '$103K - $155K (Glassdoor est.)',\n",
       "       '$93K - $192K (Glassdoor est.)', '$105K - $178K (Glassdoor est.)',\n",
       "       '$72K - $171K (Glassdoor est.)', '$101K - $180K (Glassdoor est.)',\n",
       "       '$67K - $136K (Glassdoor est.)', '$93K - $196K (Glassdoor est.)',\n",
       "       '$90K - $182K (Glassdoor est.)', '$56K - $112K (Glassdoor est.)',\n",
       "       '$84K - $173K (Glassdoor est.)', '$74K - $154K (Glassdoor est.)',\n",
       "       '$60K - $107K (Glassdoor est.)', '$89K - $182K (Glassdoor est.)',\n",
       "       '$93K - $184K (Glassdoor est.)', '$125K - $233K (Glassdoor est.)',\n",
       "       '$61K - $116K (Glassdoor est.)', '$105K - $198K (Glassdoor est.)'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"salary estimate\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observation_univariate: [0, \"salary estimate\"].\\\n",
    "  \\\n",
    "  Everything is a string specifying an interval or a single number. There are NaNs.\\\n",
    "  Some salary estimates are hourly. Some just include 1 number as an estimate.\\\n",
    "  Some include a range. The salary column is an object dtype. There are 3 different\\\n",
    "  cases: one includes \"Employer Provided Salary\", one has no range, one has the\\\n",
    "  \"(Glassdoor est.)\". We will have to build a parser for these!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build something like the HQ one where we simply remove the NaNs. This is an important statistic so we want instances that have a salary estimate. We will probably shave off a giant chunk of our data! :( But this is necessary if we want to train a predictive model. We don't want to impute our *labels*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Remove NaNs from salary.\n",
    "# 2. Parse the salary and extract average.\n",
    "\n",
    "class RemoveNaNSalary(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        return X_copy[X_copy[\"salary estimate\"] != \"-1\"].reset_index(drop=True)\n",
    "\n",
    "class SalaryParser(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,):\n",
    "        pass\n",
    "    \n",
    "    # Parse and replace the salary column with the average of the provided salary range.\n",
    "    def _salary_parser(self, salary):\n",
    "        if \"Employer Provided Salary\" in salary:\n",
    "            # salary_range can sometimes be just 1 value.\n",
    "            salary_range = re.sub(r\"[a-zA-Z]+|\\s|\\$|\\:\", \"\", salary).split(\"-\")\n",
    "        elif \"(Glassdoor est.)\" in salary:\n",
    "            salary_range = re.sub(r\"[a-zA-Z]+|\\s|\\$|\\.|[()]\", \"\", salary).split(\"-\")\n",
    "        avg_salary = np.mean(np.array(salary_range, dtype=np.float64))\n",
    "        return avg_salary\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy[\"salary estimate\"] = X_copy[\"salary estimate\"].map(self._salary_parser)\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_salary_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSalary()),\n",
    "    (\"parser\", SalaryParser())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = clean_salary_pipeline.fit_transform(df_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>209.0</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>143.0</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Harnham US</td>\n",
       "      <td>3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>190.0</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>51 to 200 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>business services</td>\n",
       "      <td>$25 to $50 million (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Abl Schools</td>\n",
       "      <td>4.1</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>140.0</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>1 to 50 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>business services</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Palo Alto, CA</td>\n",
       "      <td>115.0</td>\n",
       "      <td>Job Type : Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       company  rating       headquarters  salary estimate  \\\n",
       "0       Indeed     4.3  San Francisco, CA            209.0   \n",
       "1       Indeed     4.3  San Francisco, CA            143.0   \n",
       "2   Harnham US     3.8  San Francisco, CA            190.0   \n",
       "3  Abl Schools     4.1  San Francisco, CA            140.0   \n",
       "4       Amazon     3.8      Palo Alto, CA            115.0   \n",
       "\n",
       "               job type                 size               type  \\\n",
       "0  Job Type : Full-time     10000+ employees  company - private   \n",
       "1  Job Type : Full-time     10000+ employees  company - private   \n",
       "2        Job Type : N/A  51 to 200 employees  company - private   \n",
       "3  Job Type : Full-time    1 to 50 employees  company - private   \n",
       "4  Job Type : Full-time     10000+ employees   company - public   \n",
       "\n",
       "                   sector                   revenue  \n",
       "0  information technology    $2 to $5 billion (usd)  \n",
       "1  information technology    $2 to $5 billion (usd)  \n",
       "2       business services  $25 to $50 million (usd)  \n",
       "3       business services  unknown / non-applicable  \n",
       "4  information technology        $10+ billion (usd)  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4. job type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Job Type : Full-time', 'Job Type : N/A', 'Job Type : Part-time',\n",
       "       'Job Type : Contract', 'Job Type : Internship',\n",
       "       'Job Type : Temporary'], dtype=object)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"job type\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Harnham US</td>\n",
       "      <td>3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>190.0</td>\n",
       "      <td>Job Type : N/A</td>\n",
       "      <td>51 to 200 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>business services</td>\n",
       "      <td>$25 to $50 million (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      company  rating       headquarters  salary estimate        job type  \\\n",
       "2  Harnham US     3.8  San Francisco, CA            190.0  Job Type : N/A   \n",
       "\n",
       "                  size               type             sector  \\\n",
       "2  51 to 200 employees  company - private  business services   \n",
       "\n",
       "                    revenue  \n",
       "2  $25 to $50 million (usd)  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[df_clean[\"job type\"] == \"Job Type : N/A\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"job type\"].\\\n",
    "  \\\n",
    "  I need to RegEx parse the column values to\\\n",
    "  remove the \"Job Type :\" part. Then,\\\n",
    "  I need to replace the N/A with NaNs and finally\\\n",
    "  remove or impute the NaNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Job Type : Full-time     1913\n",
       "Job Type : Part-time       32\n",
       "Job Type : Contract         8\n",
       "Job Type : Internship       4\n",
       "Job Type : Temporary        2\n",
       "Job Type : N/A              1\n",
       "Name: job type, dtype: int64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"job type\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       Job Type : Full-time\n",
       "1       Job Type : Full-time\n",
       "2             Job Type : N/A\n",
       "3       Job Type : Full-time\n",
       "4       Job Type : Full-time\n",
       "                ...         \n",
       "1955    Job Type : Full-time\n",
       "1956    Job Type : Full-time\n",
       "1957    Job Type : Full-time\n",
       "1958    Job Type : Full-time\n",
       "1959    Job Type : Full-time\n",
       "Name: job type, Length: 1960, dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"job type\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the N/A job types and also parse them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RemoveNaNJobType(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        return X_copy[X_copy[\"job type\"].apply(lambda x: x.split(\" \")[-1] != \"N/A\")].reset_index(drop=True)\n",
    "        \n",
    "        \n",
    "class JobTypeParser(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self,):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy[\"job type\"] = X_copy[\"job type\"].map(lambda x: x.split()[-1])\n",
    "        \n",
    "        return X_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_jobtype_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNJobType()),\n",
    "    (\"parser\", JobTypeParser())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = clean_jobtype_pipeline.fit_transform(df_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Full-time', 'Part-time', 'Contract', 'Internship', 'Temporary'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"job type\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1959, 9)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5. size, founded, type, industry, sector, revenue, and job description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000+ employees           596\n",
       "1 to 50 employees          347\n",
       "51 to 200 employees        267\n",
       "1001 to 5000 employees     196\n",
       "201 to 500 employees       188\n",
       "501 to 1000 employees      168\n",
       "5001 to 10000 employees    126\n",
       "unknown                     71\n",
       "Name: size, dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"size\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"size\"].\\\n",
    "  \\\n",
    "  There are 71 unknown values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "company - public                  886\n",
       "company - private                 856\n",
       "subsidiary or business segment     56\n",
       "nonprofit organization             52\n",
       "college / university               37\n",
       "unknown                            26\n",
       "contract                           15\n",
       "government                         10\n",
       "self-employed                       9\n",
       "hospital                            6\n",
       "private practice / firm             5\n",
       "school / school district            1\n",
       "Name: type, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"type\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"type\"].\\\n",
    "  \\\n",
    "  26 unknowns, not too many."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "information technology                863\n",
       "-1                                    241\n",
       "biotech & pharmaceuticals             187\n",
       "business services                     139\n",
       "finance                               105\n",
       "retail                                 97\n",
       "health care                            61\n",
       "manufacturing                          58\n",
       "education                              48\n",
       "insurance                              43\n",
       "oil, gas, energy & utilities           26\n",
       "non-profit                             20\n",
       "media                                  18\n",
       "transportation & logistics             12\n",
       "government                             11\n",
       "real estate                             7\n",
       "accounting & legal                      6\n",
       "telecommunications                      6\n",
       "restaurants, bars & food services       4\n",
       "construction, repair & maintenance      3\n",
       "arts, entertainment & recreation        2\n",
       "agriculture & forestry                  1\n",
       "travel & tourism                        1\n",
       "Name: sector, dtype: int64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"sector\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"sector\"].\\\n",
    "  \\\n",
    "  Lots of missing values and a few obscure unique values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unknown / non-applicable            855\n",
       "$10+ billion (usd)                  475\n",
       "$2 to $5 billion (usd)               94\n",
       "$100 to $500 million (usd)           94\n",
       "$1 to $2 billion (usd)               86\n",
       "less than $1 million (usd)           76\n",
       "$5 to $10 billion (usd)              53\n",
       "$25 to $50 million (usd)             50\n",
       "$50 to $100 million (usd)            44\n",
       "$10 to $25 million (usd)             44\n",
       "$1 to $5 million (usd)               43\n",
       "$500 million to $1 billion (usd)     28\n",
       "$5 to $10 million (usd)              17\n",
       "Name: revenue, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[\"revenue\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 🔍 observations_univariate: [0, \"revenue\"].\\\n",
    "  \\\n",
    "  Nearly half of the records don't have a recorded revenue. We can't afford to drop it, we will leave it in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RemoveNaNSize(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"size\"] == \"-1\"])\n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"size\"] == \"unknown\"])\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)\n",
    "    \n",
    "class RemoveNaNType(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"type\"] == \"-1\"])\n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"type\"] == \"unknown\"])\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)\n",
    "    \n",
    "class RemoveNaNSector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"sector\"] == \"-1\"])\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)\n",
    "    \n",
    "class RemoveNaNRevenue(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X_copy = X.copy()\n",
    "        \n",
    "        X_copy = X_copy.drop(X_copy.index[X_copy[\"revenue\"] == \"-1\"])\n",
    "        \n",
    "        return X_copy.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_size_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSize())\n",
    "])\n",
    "\n",
    "clean_type_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNType())\n",
    "])\n",
    "\n",
    "clean_sector_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSector())\n",
    "])\n",
    "\n",
    "clean_revenue_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNRevenue())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = clean_size_pipeline.fit_transform(df_clean)\n",
    "df_clean = clean_type_pipeline.fit_transform(df_clean)\n",
    "df_clean = clean_sector_pipeline.fit_transform(df_clean)\n",
    "df_clean = clean_revenue_pipeline.fit_transform(df_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1701, 9)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>209.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>143.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abl Schools</td>\n",
       "      <td>4.1</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>140.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>1 to 50 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>business services</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Palo Alto, CA</td>\n",
       "      <td>115.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America</td>\n",
       "      <td>3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>134.5</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   company  rating       headquarters  salary estimate  \\\n",
       "0                   Indeed     4.3  San Francisco, CA            209.0   \n",
       "1                   Indeed     4.3  San Francisco, CA            143.0   \n",
       "2              Abl Schools     4.1  San Francisco, CA            140.0   \n",
       "3                   Amazon     3.8      Palo Alto, CA            115.0   \n",
       "4  Thermo Fisher - America     3.8  San Francisco, CA            134.5   \n",
       "\n",
       "    job type               size               type                     sector  \\\n",
       "0  Full-time   10000+ employees  company - private     information technology   \n",
       "1  Full-time   10000+ employees  company - private     information technology   \n",
       "2  Full-time  1 to 50 employees  company - private          business services   \n",
       "3  Full-time   10000+ employees   company - public     information technology   \n",
       "4  Full-time   10000+ employees   company - public  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue  \n",
       "0    $2 to $5 billion (usd)  \n",
       "1    $2 to $5 billion (usd)  \n",
       "2  unknown / non-applicable  \n",
       "3        $10+ billion (usd)  \n",
       "4        $10+ billion (usd)  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.6. Building the Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparams.\n",
    "remove_cols = [\"job title\", \"founded\", \"job description\", \"industry\"]\n",
    "\n",
    "replace_company_names = {\"Amazon\": \"Amazon\", \n",
    "                         \"Parker Institute\": \"Parker Institute for Cancer Immunotherapy\", \n",
    "                         \"Sony\": \"Sony\", \n",
    "                         \"Stealth Mode\": \"Stealth Mode\", \n",
    "                         \"Twitch\": \"Twitch\", \n",
    "                         \"Wells Fargo\": \"Wells Fargo\", \n",
    "                         \"Varo\": \"Varo\"}\n",
    "\n",
    "remove_company_names = [\"financial services company\", \"information technology sector\"]\n",
    "\n",
    "# Pipelines.\n",
    "clean_general_pipeline = Pipeline([\n",
    "    (\"nan_row_remover\", RemoveNaNRows()),\n",
    "    (\"col_selector\", DataFrameSelector(remove_cols, include=False)),\n",
    "    (\"index_remover\", IndexRemover())\n",
    "])\n",
    "\n",
    "clean_company_pipeline = Pipeline([\n",
    "    (\"company_rating_split\", CompanyRatingSplit()),\n",
    "    (\"company_aggregate\", AggregateCompanyNames(replace_company_names)),\n",
    "    (\"company_remove\", RemoveNonCompanyNames(remove_company_names))\n",
    "])\n",
    "\n",
    "clean_rating_pipeline = Pipeline([\n",
    "    (\"rating_fill_by_cmpny_avg\", FillRatingByCmpnyAvg()),\n",
    "    (\"rating_fill_by_avg\", FillRatingByAvg())\n",
    "])\n",
    "\n",
    "clean_hq_pipeline = Pipeline([\n",
    "    (\"remove_ext_hq\", RemoveExtraneousHQ())\n",
    "])\n",
    "\n",
    "clean_salary_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSalary()),\n",
    "    (\"parser\", SalaryParser())\n",
    "])\n",
    "\n",
    "clean_jobtype_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNJobType()),\n",
    "    (\"parser\", JobTypeParser())\n",
    "])\n",
    "\n",
    "clean_size_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSize())\n",
    "])\n",
    "\n",
    "clean_type_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNType())\n",
    "])\n",
    "\n",
    "clean_sector_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNSector())\n",
    "])\n",
    "\n",
    "clean_revenue_pipeline = Pipeline([\n",
    "    (\"nan_remover\", RemoveNaNRevenue())\n",
    "])\n",
    "\n",
    "cleaning_pipeline = Pipeline([\n",
    "    (\"general\", clean_general_pipeline),\n",
    "    (\"company\", clean_company_pipeline),\n",
    "    (\"rating\", clean_rating_pipeline),\n",
    "    (\"headquarters\", clean_hq_pipeline),\n",
    "    (\"salary\", clean_salary_pipeline),\n",
    "    (\"jobtype\", clean_jobtype_pipeline),\n",
    "    (\"size\", clean_size_pipeline),\n",
    "    (\"type\", clean_type_pipeline),\n",
    "    (\"sector\", clean_sector_pipeline),\n",
    "    (\"revenue\", clean_revenue_pipeline),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = cleaning_pipeline.fit_transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1701, 9)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company</th>\n",
       "      <th>rating</th>\n",
       "      <th>headquarters</th>\n",
       "      <th>salary estimate</th>\n",
       "      <th>job type</th>\n",
       "      <th>size</th>\n",
       "      <th>type</th>\n",
       "      <th>sector</th>\n",
       "      <th>revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>209.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Indeed</td>\n",
       "      <td>4.3</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>143.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$2 to $5 billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abl Schools</td>\n",
       "      <td>4.1</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>140.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>1 to 50 employees</td>\n",
       "      <td>company - private</td>\n",
       "      <td>business services</td>\n",
       "      <td>unknown / non-applicable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>3.8</td>\n",
       "      <td>Palo Alto, CA</td>\n",
       "      <td>115.0</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>information technology</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thermo Fisher - America</td>\n",
       "      <td>3.8</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>134.5</td>\n",
       "      <td>Full-time</td>\n",
       "      <td>10000+ employees</td>\n",
       "      <td>company - public</td>\n",
       "      <td>biotech &amp; pharmaceuticals</td>\n",
       "      <td>$10+ billion (usd)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   company  rating       headquarters  salary estimate  \\\n",
       "0                   Indeed     4.3  San Francisco, CA            209.0   \n",
       "1                   Indeed     4.3  San Francisco, CA            143.0   \n",
       "2              Abl Schools     4.1  San Francisco, CA            140.0   \n",
       "3                   Amazon     3.8      Palo Alto, CA            115.0   \n",
       "4  Thermo Fisher - America     3.8  San Francisco, CA            134.5   \n",
       "\n",
       "    job type               size               type                     sector  \\\n",
       "0  Full-time   10000+ employees  company - private     information technology   \n",
       "1  Full-time   10000+ employees  company - private     information technology   \n",
       "2  Full-time  1 to 50 employees  company - private          business services   \n",
       "3  Full-time   10000+ employees   company - public     information technology   \n",
       "4  Full-time   10000+ employees   company - public  biotech & pharmaceuticals   \n",
       "\n",
       "                    revenue  \n",
       "0    $2 to $5 billion (usd)  \n",
       "1    $2 to $5 billion (usd)  \n",
       "2  unknown / non-applicable  \n",
       "3        $10+ billion (usd)  \n",
       "4        $10+ billion (usd)  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1701 entries, 0 to 1700\n",
      "Data columns (total 9 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   company          1701 non-null   object \n",
      " 1   rating           1701 non-null   float64\n",
      " 2   headquarters     1701 non-null   object \n",
      " 3   salary estimate  1701 non-null   float64\n",
      " 4   job type         1701 non-null   object \n",
      " 5   size             1701 non-null   object \n",
      " 6   type             1701 non-null   object \n",
      " 7   sector           1701 non-null   object \n",
      " 8   revenue          1701 non-null   object \n",
      "dtypes: float64(2), object(7)\n",
      "memory usage: 119.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df_clean.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Saving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will save 2 copies of our cleaned csv. One without the \"job title\" and \"job description\" columns and one with them. The one without will be easy to preprocess and the one with these 2 additional columns will be great for EDA!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean dataframe with \"job title\" and \"job description\" columns included.\n",
    "\n",
    "remove_cols = [\"founded\", \"industry\"]\n",
    "\n",
    "clean_general_pipeline = Pipeline([\n",
    "    (\"nan_row_remover\", RemoveNaNRows()),\n",
    "    (\"col_selector\", DataFrameSelector(remove_cols, include=False)),\n",
    "    (\"index_remover\", IndexRemover())\n",
    "])\n",
    "\n",
    "\n",
    "cleaning_pipeline = Pipeline([\n",
    "    (\"general\", clean_general_pipeline),\n",
    "    (\"company\", clean_company_pipeline),\n",
    "    (\"rating\", clean_rating_pipeline),\n",
    "    (\"headquarters\", clean_hq_pipeline),\n",
    "    (\"salary\", clean_salary_pipeline),\n",
    "    (\"jobtype\", clean_jobtype_pipeline),\n",
    "    (\"size\", clean_size_pipeline),\n",
    "    (\"type\", clean_type_pipeline),\n",
    "    (\"sector\", clean_sector_pipeline),\n",
    "    (\"revenue\", clean_revenue_pipeline),\n",
    "])\n",
    "\n",
    "df_clean_with_text = cleaning_pipeline.fit_transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean.to_csv(\"../input/joblisting_cleaned.csv\", index=False)\n",
    "df_clean.to_csv(\"./input/joblisting_cleaned.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean_with_text.to_csv(\"../input/joblisting_cleaned_withtext.csv\", index=False)\n",
    "df_clean_with_text.to_csv(\"./input/joblisting_cleaned_withtext.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}