Job_word_embeddings_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this notebook, we will implement word embeddings for job postings using [Word2vec](https://en.wikipedia.org/wiki/Word2vec). By implementing this, you will learn about embedding words in job postings such as job titles and skills. This will be useful for semantic job search, for instance.\n",
    "\n",
    "Table of contents\n",
    "\n",
    "- [Word embeddings and word2vec](#Word-embeddings,-word2vec)\n",
    "- [Job posting data](#Job-posting-data)\n",
    "- [Data preprocessing](#Data-preprocessing)\n",
    "- [Training](#Training)\n",
    "- [Visualization](#Visualization)\n",
    "- [Future work](#Future-work)\n",
    "- [Reading](#Reading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word embeddings and word2vec\n",
    "Word2vec is a machine learning model used for learning vector representation of words, called \"word embeddings\".\n",
    "Before starting implemenation, let's look at why we would want to learn word embeddings in the first place."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why learn word embeddings?\n",
    "Natural language processing (NLP) traditonally convert words to discrete symbols or ids.  \n",
    "e.g. `I like cats and dogs.` -> `[23, 761, 748, 221, 309] `\n",
    "\n",
    "These encondings provide no useful information about relationships that may exist between individual symbols. Furthermore, representing words as unique, discrete ids leads to data sparsity, and requires a lot of data.  \n",
    "Using vector representations can overcome some of these issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2vec\n",
    "Word2vec is a neural network model which allows you to vectorize words.\n",
    "If you are interested in how to calculate vectors of words, check out the readings listed below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Architectures\n",
    "There are 2 architectures for implementing word2vec, **CBOW** (Continuous Bag-Of-Words) and **Skip-gram**.  \n",
    "\n",
    "**CBOW** is learning to predict the word by the context, on the other hand, the **Skip-gram** is designed to predict the context.\n",
    "For example, if the sentence is \"*The cat ate the mouse*\"  and the current word is `ate`,   \n",
    "In **CBOW**, `['The', 'cat', 'the', 'mouse']` will be calculating `ate`, while `ate` will be calculating `['The', 'cat', 'the', 'mouse']` in **Skip-gram**.\n",
    "\n",
    "<img src=\"assets/word2vec_diagrams.png\" width=\"700\">\n",
    "\n",
    "You don't need to understand these architectures completely, the important point is that the word2vec model learns the mearning of words from contexts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity\n",
    "Each word will be located as a point in hundreds-dimentional vectorspace (practically 100 - 500 dimentions).  \n",
    "A well trained set of word vectors will place similar words close to each other in that space.  \n",
    "For instance, `dogs` and `cats` might cluster in one corner, while `war`, `conflict` and `strife` in another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translation\n",
    "Word2vec can learn many associations other than similarity.\n",
    "For instance, it can gauge relations between words of one language, and map them to another.\n",
    "\n",
    "<img src=\"assets/word2vec_translation.png\" width=\"700\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance\n",
    "These vectors are the basis of a more comprehensive geometry of words.  Not only will Rome, Paris, Berlin and Beijing cluster near each other, but they will each have similar distances in vectorspace to the countries whose capitals they are.\n",
    "\n",
    "<img src=\"assets/countries_capitals.png\" width=\"700\">\n",
    "\n",
    "So you can find the capital of Japan by calculating `Rome - Italy + Japan = Tokyo`.  \n",
    "Another famous example would be `king - man + woman = queen`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How are job word embeddings useful?\n",
    "One of the reasons why we would want to learn \"job word embeddings\" is that it will be useful when finding jobs that have simiar job titles with your current one, which is difficult to be done by traditonal search engine.  \n",
    "\n",
    "<style>\n",
    "table {margin: none;}\n",
    "</style>\n",
    "\n",
    "|Target|Similar job titles|\n",
    "|:---|:---|\n",
    "|Web engineer| Server side eingeer, fullstack engineer, front-end engineer|\n",
    "|Infra engineer| Devops engineer, System reliability engineer|\n",
    "|Data scientist| Data analyist, Data engineer, Machine learning engineer|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Job posting data\n",
    "We need job postings text data to learn word embeddings.\n",
    "\n",
    "The prepared dataset has 12130 postings with many fields: `job_title`, `summary`, `requirements`, `salary`, `location` etc. some explain job content well, other don't. We can decide which fields to use for learning.\n",
    "\n",
    "Let's look at the raw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "job_postings = pd.read_csv('./data/job_postings.csv')\n",
    "job_postings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preprocessing\n",
    "\n",
    "In this project, we will use `requirements`, `summary` as they seem to be explaining job contents.  and also append `job_title` to learn what each job title means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_job = job_postings.loc[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_job[['requirements', 'summary']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function extracts keywords from these fields.  \n",
    "We will use only nouns because adjectives, verbs and marks don't give meaningful information for this material."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import MeCab\n",
    "# check where neologd is in your computer, and change the path in the line below.\n",
    "tagger = MeCab.Tagger(\"-U %m,未知語\\\\t -F %f[0],%f[6]\\\\t  -d  /usr/local/opt/mecab-ipadic-neologd\")\n",
    "\n",
    "def extract_nouns(text): \n",
    "    words = []\n",
    "    for row in tagger.parse(text).split('\\t'):\n",
    "        if row.strip() == 'EOS':\n",
    "            continue\n",
    "        t = row.split(',')[0]\n",
    "        w = row.split(',')[1]\n",
    "        if t == '名詞':\n",
    "            words.append(w.strip().lower())\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(extract_nouns(sample_job['requirements']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that there are words that explains the job such as `web`, `application`, `java`, `javascript` etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(extract_nouns(sample_job['summary']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the summary field, there are words related to the job like `java`, `swift`, `backlog` etc, but also some words that represent the business of the company like `crm`, `no.1`, `platform`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were able to extract meaningful words from the `requirements` and `summary` fields.  \n",
    "Let's check `job_title` field next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the first 20 job titles\n",
    "job_postings['job_title'].head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to use job titles without tokenization to learn what each job title means.  \n",
    "e.g. web engineer, web application engineer, lead engineer . \n",
    "\n",
    "But as you can see above, the raw data are written in a variety of ways.  \n",
    "So you need to clean them up to normalize job titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import unicodedata\n",
    "\n",
    "job_titles = job_postings['job_title']\n",
    "\n",
    "# Remove brackets from text\n",
    "# e.g. インフラエンジニア(ADPLAN) -> インフラエンジニア\n",
    "def normalize_job_title(title):\n",
    "    title = unicodedata.normalize('NFKC', title)\n",
    "    title = re.sub(r'【.*】', '', title)\n",
    "    title = re.sub(r'\\[.*\\]', '', title)\n",
    "    title = re.sub(r'「.*」', '', title)\n",
    "    title = re.sub(r'\\(.*\\)', '', title)\n",
    "    title = re.sub(r'\\<.*\\>', '', title)\n",
    "    title = re.sub(r'[※@◎].*$', '', title)\n",
    "    return title.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply the function to the first 20 job titles\n",
    "job_postings['job_title'].head(20).apply(normalize_job_title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coverting to the input of word2vec API\n",
    "\n",
    "In the next section, we will use [Gensim's word2vec API](https://radimrehurek.com/gensim/models/word2vec.html) to produce word embeddings.  \n",
    "As you can see in the documentation, it requires lists of words.\n",
    "\n",
    "```python\n",
    "from gensim.models import Word2Vec\n",
    "sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\n",
    "\n",
    "model = Word2Vec(sentences)\n",
    "say_vector = model['say']  # get vector for word\n",
    "```\n",
    "\n",
    "So we need to convert the raw data to the required format. You can do that with the functions implemented above.  \n",
    "You also need to append job titles to get mearning out of them.  \n",
    "You can do that by adding them to the beginning of requirements.\n",
    "\n",
    "```python\n",
    "['web engineer'] + ['ruby', 'postgresql', 'agile', 'developement', ... ]\n",
    "```\n",
    "\n",
    "Now, it's your time to get your hands dirty and prepare input data for training.  \n",
    "\n",
    "**Exercise**: Complete the function `convert_job_posting` and convert all job posting to appropriate format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_job_posting(job):\n",
    "    # implement this function\n",
    "    pass\n",
    "\n",
    "# convert data to inputs of Gensim's word2vec API\n",
    "inputs = []\n",
    "for _, p in job_postings.iterrows():\n",
    "    inputs += convert_job_posting(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(inputs[20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "Finally, you can learn word embeddings with the inputs you prepared above and gensim's word2vec.\n",
    "\n",
    "**Exercise**: build word2vec model checking [the API document](https://radimrehurek.com/gensim/models/word2vec.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.word2vec  import Word2Vec\n",
    "\n",
    "word2vec_model = # build model\n",
    "\n",
    "word2vec_model.save('job_word_embeddings.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word2vec model has a method called `most_similar` to return similar words from learned corpus.  \n",
    "You can use a wrapper function defined beflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def similar_words(title):\n",
    "    return word2vec_model.most_similar(title.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try finding similar words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('Webエンジニア') # web engineer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('UIデザイナー') # UIデザイナー"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('aiエンジニア') # ai engineer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('Ruby')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('photoshop')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_words('agile')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results will be useful when finding jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization\n",
    "\n",
    "We looked at similar words in the previous section, but the learned model has not only similarities, but also geometric information.  \n",
    "The following code will plot the vectors of the most popular 1000 job titles in 2-dimentional space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "from matplotlib.font_manager import FontProperties\n",
    "from sklearn.manifold import TSNE\n",
    "import numpy as np\n",
    "\n",
    "# Check the Japanese font file in your machine, and change the line below.\n",
    "fp = FontProperties(fname='/System/Library/Fonts/Hiragino Sans GB W3.ttc', size=14)\n",
    "\n",
    "popular_job_titles = job_postings['job_title'].map(normalize_job_title).value_counts()[0:1000].keys()\n",
    "emb_tuple = tuple([word2vec_model[t] for t in popular_job_titles if t in word2vec_model] )\n",
    "X = np.vstack(emb_tuple)\n",
    "\n",
    "model = TSNE(n_components=2, random_state=0)\n",
    "np.set_printoptions(suppress=True)\n",
    "model.fit_transform(X) \n",
    "\n",
    "plt.figure(figsize=(40,40))\n",
    "plt.scatter(model.embedding_[:, 0], model.embedding_[:, 1])\n",
    "\n",
    "for label, x, y in zip(popular_job_titles, model.embedding_[:, 0], model.embedding_[:, 1]):\n",
    "    plt.annotate(label, xy=(x, y), xytext=(0, 0), textcoords='offset points', fontproperties=fp)\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see similar job titles are plotted closely.  \n",
    "This information will be useful when clustering job titles or skills. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional exercises \n",
    "I suggest you to try further exercises if you are intertested.\n",
    "\n",
    "## Parameter tuning\n",
    "In the above exercise, you didn't change the training parameters. \n",
    "\n",
    "```python\n",
    "word2vec.Word2Vec(inputs, size=100, min_count=5, window=10, sg=1)\n",
    "```\n",
    "\n",
    "Reading [the gensim's document](https://radimrehurek.com/gensim/models/word2vec.html), change those parameters and see how it affects the output.\n",
    "\n",
    "\n",
    "## Clustering\n",
    "Find clusters from vectorized words.  \n",
    "Scikit-learn provides[Kmean clustering API](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html). [This article](http://ai.intelligentonlinetools.com/ml/k-means-clustering-example-word2vec/) explains how to use with sample code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Future work\n",
    "In this final section, we will see potential applications for job word embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic job search\n",
    "One useful application would be \"semantic\" job search with which you can find jobs not only by your query, but also by similar words with it.  \n",
    "The idea is not complicated. the system converts query to original query + similar words (some of you may remember the project :P )  \n",
    "It may be easilly implemented by using some of the Elasticsearch features.\n",
    "\n",
    "- [Synonym Token Filter](https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-synonym-tokenfilter.html)\n",
    "  - You can register similar words as synonyms\n",
    "- [Compound Queries](https://www.elastic.co/guide/en/elasticsearch/reference/master/compound-queries.html)\n",
    "  - Needs more investigation, but you can probably use similarity scores as weights of search score.\n",
    "\n",
    "There are some articles/projects that do the same thing.\n",
    "\n",
    "- [Cocenptual Job Search](https://github.com/DiceTechJobs/ConceptualSearch)\n",
    "- [Using Word2Vec in Fusion for Better Search Results](https://lucidworks.com/2016/11/16/word2vec-fusion-nlp-search/)\n",
    "- [ThisPlusThat: A Search Engine That Lets You ‘Add’ Words as Vectors](https://blog.insightdatascience.com/thisplusthat-me-a-search-engine-that-lets-you-add-words-as-vectors-2ec0b8a4f629)\n",
    "\n",
    "## Sematic candidate search\n",
    "It's similar to job search. but it's also difficult for recuters to find good candidates, expecially when there are not many candidates who satifsfy the requirements of a position. e.g. data scientist, creative director, product owner, senior engineer etc.  \n",
    "Using similar job titles and skills, you can find more candidates semantically.\n",
    "\n",
    "## Similarity-based job recommendation system\n",
    "There is an advanced model that vectorizes documents, called \"[doc2vec](https://radimreoshurek.com/gensim/models/doc2vec.html)\". When a user has already applied to or bookmarked some job positions, you can recommend similar jobs with those prefered ones.\n",
    "\n",
    "- [Recommender System with Distributed Representation](https://www.slideshare.net/rakutentech/recommender-system-with-distributed-representation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading\n",
    "Here are some resources which you can learn more about word2vec after workshop.\n",
    "- [Word2vec explanation in DL4j document](https://deeplearning4j.org/word2vec)\n",
    "- [Gensim documentation](https://radimrehurek.com/gensim/models/word2vec.html)\n",
    "- [Gensim word2vec tutorial](https://rare-technologies.com/word2vec-tutorial/)\n",
    "- [Tensorflow tutorial](https://www.tensorflow.org/tutorials/word2vec)\n",
    "- [Udacity material](https://github.com/udacity/deep-learning/blob/master/embeddings/Skip-Gram_word2vec.ipynb)\n",
    "- [Conceptual overview](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}