models/model_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gilda disambiguation model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates, through a specific example, how to train a new Gilda disambiguation model. We will use the example of \"DOM\". \n",
    "\n",
    "Below, we demonstrate the nature of this ambiguity: we ground the string \"DOM\" with Gilda and find that both a human gene (whose symbol is SOX10) and a small molecule listed by MeSH (whose standard name is 2,5-Dimethoxy-4-Methylamphetamine) have this as a synonym."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "ambig_text = 'DOM'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredMatch(Term(dom,DOM,HGNC,11190,SOX10,synonym,hgnc,9606,None,None),0.5555555555555556,Match(query=DOM,ref=DOM,exact=True,space_mismatch=False,dash_mismatches={},cap_combos=[])),\n",
       " ScoredMatch(Term(dom,DOM,MESH,D004290,2,5-Dimethoxy-4-Methylamphetamine,synonym,mesh,None,None,None),0.5555555555555556,Match(query=DOM,ref=DOM,exact=True,space_mismatch=False,dash_mismatches={},cap_combos=[]))]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gilda import ground\n",
    "matches = ground(ambig_text)\n",
    "matches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1. Retrieve PMIDs for each sense of the ambiguous string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train a model to disambiguate DOM, we need to first obtain a set of unique PMIDs for both the human gene and the MeSH term. We use the PubMed API, accessed through INDRA, to perform this search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from indra.literature import pubmed_client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "terms = [match.term for match in matches]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_pmids = {}\n",
    "for term in terms:\n",
    "    if term.db == 'HGNC':\n",
    "        pmids = pubmed_client.get_ids_for_gene(term.entry_name)\n",
    "    elif term.db == 'MESH':\n",
    "        pmids = pubmed_client.get_ids_for_mesh(term.id, major_topic=True)\n",
    "    term_pmids[(term.db, term.id)] = pmids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We got 210 PMIDs for HGNC:11190.\n",
      "We got 194 PMIDs for MESH:D004290.\n"
     ]
    }
   ],
   "source": [
    "for (term_db, term_id), pmids in term_pmids.items():\n",
    "    print('We got %d PMIDs for %s:%s.' % (len(pmids), term_db, term_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generally don't want to take PMIDs that overlap between multiple terms since these are not useful for disambiguation between them. In this case, there aren't any overlapping PMIDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(term_pmids[('HGNC', '11190')]) & set(term_pmids[('MESH', 'D004290')]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Download text content for each PMID for each sense of the ambiguous string "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next need to obtain text content for these PMIDs which will become the actual labeled training data. We can do this again through the PubMed API. For large scale training (i.e., training a large number of models), it makes sense to build a local cache of text content first for faster retrieval. Additionally, if full text content is available for a given paper (through e.g., PubMedCentral), it can be used here instead of abstracts. For simplicity, here we simply use abstracts from the PubMed API which take a few minutes to retrieve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Getting text content for HGNC:11190: 100%|██████████| 210/210 [04:34<00:00,  1.31s/it]\n",
      "Getting text content for MESH:D004290: 100%|██████████| 194/194 [04:09<00:00,  1.29s/it]\n"
     ]
    }
   ],
   "source": [
    "import tqdm\n",
    "import time\n",
    "from collections import defaultdict\n",
    "term_texts = defaultdict(list)\n",
    "for (term_db, term_id), pmids in term_pmids.items():\n",
    "    for pmid in tqdm.tqdm(pmids, desc='Getting text content for %s:%s' % (term_db, term_id)):\n",
    "        txt = pubmed_client.get_abstract(pmid, prepend_title=True)\n",
    "        if txt:\n",
    "            term_texts[(term_db, term_id)].append(txt)\n",
    "        # The PubMed API doesn't allow very frequent calls so we add a bit of delay\n",
    "        time.sleep(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train a disambiguation model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all our text content, we can set up the model training. For this, we put our text content into a single flat list, and create a labels list that provides a label identifying the corresponding entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts_list = []\n",
    "labels_list = []\n",
    "for (term_db, term_id), texts in term_texts.items():\n",
    "    texts_list += texts\n",
    "    labels_list += ['%s:%s' % (term_db, term_id)]*len(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now instantiate the classifier from Adeft and train it through cross-validation. We then print model statistics showing various metrics of model quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: [2022-03-16 23:18:21] /Users/ben/Dropbox/postdoc/darpa/src/deft/adeft/modeling/classify.py - Beginning grid search in parameter space:\n",
      "{'C': [10.0], 'max_features': [100, 1000], 'ngram_range': [(1, 2)]}\n",
      "INFO: [2022-03-16 23:18:23] /Users/ben/Dropbox/postdoc/darpa/src/deft/adeft/modeling/classify.py - Best f1 score of 0.9974999999999999 found for parameter values:\n",
      "{'logit__C': 10.0, 'tfidf__max_features': 1000, 'tfidf__ngram_range': (1, 2)}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'label_distribution': {'HGNC:11190': 207, 'MESH:D004290': 194}, 'f1': {'mean': 0.9975, 'std': 0.005}, 'precision': {'mean': 0.9975, 'std': 0.005}, 'recall': {'mean': 0.9975, 'std': 0.005}, 'HGNC:11190': {'f1': {'mean': 0.997531, 'std': 0.004938}, 'pr': {'mean': 0.995122, 'std': 0.009756}, 'rc': {'mean': 1.0, 'std': 0.0}}, 'MESH:D004290': {'f1': {'mean': 0.997468, 'std': 0.005063}, 'pr': {'mean': 1.0, 'std': 0.0}, 'rc': {'mean': 0.995, 'std': 0.01}}}\n"
     ]
    }
   ],
   "source": [
    "from adeft.modeling.classify import AdeftClassifier\n",
    "model = AdeftClassifier([ambig_text], list(set(labels_list)))\n",
    "params = {'C': [10.0], 'max_features': [100, 1000], 'ngram_range': [(1, 2)]}\n",
    "model.cv(texts_list, labels_list, params, cv=5)\n",
    "print(model.stats)\n",
    "model_json = model.get_model_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Try out the model and save it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is now ready and we can try it out on some sample sentences (though the text provided can be much longer, including e.g., an abstract or a full text paper). We can see that different senses are predicted for the first vs the second sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'HGNC:11190': 0.7254491363594628, 'MESH:D004290': 0.27455086364053716}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_proba(['The DOM gene belongs to a family of genes that plays a critical role in the formation of tissues.'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'HGNC:11190': 0.0495657056566603, 'MESH:D004290': 0.9504342943433397}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_proba(['DOM is a psychedelic and a substituted amphetamine.'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gilda's disambiguation models are saved in a gzipped pickle file containing a Python dict where each entry has the following structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "models = {'DOM': (terms, {'cl': model_json})}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each key in the models dictionary is the ambiguous string the model applies to, and each value is a tuple. The first element of the value tuple is the list of Terms that the model was trained for (this is just there for traceability and not strictly necessary), and the second element of the value tuple is a dict containing the model in a serialized form. If a Python data structure with the same structure as `models` above is saved into `~/.data/gilda/<versions>/gilda_models.pkl.gz` (i.e., pickle dumped and then gzipped), Gilda will automatically load it and use it for disambiguation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}