01-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The ContentMine Toolkit\n",
    "\n",
    "## Introduction\n",
    "\n",
    "These notebooks aim to provide an introduction to some of the tools in the [ContentMine](http://contentmine.org/) toolkit.\n",
    "\n",
    "<p style=\"text-align: center; background-color: lightyellow;\">\n",
    "    This <a href=\"https://jupyter.org/\">Juypter Notebook</a> provide an interactive environment where you can run Python scripts and shell commands and see the results in your browser.<br/>\n",
    "    For shell commands, each line starts with a <tt>!</tt>, so <tt>!echo hello</tt> runs a shell script, but <tt>print('hello')</tt> runs as Python.<br/>\n",
    "    For a more detailed introduction, try this <a href=\"https://programminghistorian.org/en/lessons/jupyter-notebooks\">Introduction to Jupyter Notebooks</a>.\n",
    "</p>\n",
    "            \n",
    "### The Big Picture\n",
    "\n",
    "The vison that drives ContentMine is that text and data mining should be used to open up all research literature, allowing the factual content to be made available to all, and helping researchers to make better use of what has already been discovered.\n",
    "\n",
    "### Overall Approach\n",
    "\n",
    "The high-level workflow for meeting this vision works as follows.\n",
    "\n",
    "Given a research question or area of interest:\n",
    "\n",
    "- Collect all relevant publications:\n",
    "    - This may be as PDF files or as XML or HTML.\n",
    "- Normalise the publications into a better machine-readable form:\n",
    "    - As well-structured HTML, referred to here as 'scholarly HTML' (but note that this does _not_ refer to [the proposed Scholarly HTML standard](https://w3c.github.io/scholarly-html/)). Any properly marked-up HTML should work fine, but ideally re-using the subset of [JATS](https://jats.nlm.nih.gov/) tags that make sense in a HTML context.\n",
    "    - If HTML isn't an option, plain text can be used.\n",
    "    - However, depending on the question at hand, other formats might be required.\n",
    "- Extract facts:\n",
    "    - Which papers mention which terms and concepts?\n",
    "    - What research cited in each paper?\n",
    "    - Which chemical compounds appear in the diagrams of each paper.\n",
    "- Share the results:\n",
    "    - Make the normalised publications available (if licensing permits).\n",
    "    - Make the extracted facts openly available (factual assertions and [non-consumptive datasets](https://www.hathitrust.org/htrc_ncup) can usually be made available without restriction).\n",
    "    - Create visualisations and documentation to help understand the results.\n",
    "- Make new knowledge:\n",
    "    - Use the results as the foundation for your own research.\n",
    "\n",
    "Here, however, we'll focus on a specific use case and use that to explore how the approach works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The openVirus Project\n",
    "\n",
    "In response to the COVID-19 epidemic, the [openVirus project](https://github.com/petermr/openVirus#openvirus) aims to aggregate scholarly publications and extracted knowledge on viruses and epidemics.\n",
    "\n",
    "Here, we'll look at the specific case of finding which open access electronic theses mention terms relating to viruses and epidemics. Information about UK e-theses has been gathered by the [EThOS service](https://ethos.bl.uk/), and a suitable source dataset has been made available [here](https://data.bl.uk/ethos/).\n",
    "\n",
    "The specific workflow here is:\n",
    "\n",
    "- Use the EThOS dataset to find open-access theses.\n",
    "- Get the PDFs and assemble them into the file folder layout conventions of the ContentMine toolkit.\n",
    "- Generate HTML or text versions of the PDFs.\n",
    "- Search through the HTML for the relevant terms, and extract the terms along with snippets of text as context for each time the terms appear.\n",
    "\n",
    "The openVirus project has already created some [dictionaries](https://github.com/petermr/openVirus/tree/master/dictionaries) that can be used to link the terms that appear in the text to the relevant WikiData entities. _TBA come back to dictionary creation later on_\n",
    "\n",
    "But to get started, we need some data sources. We'll start with a single theses, found by searching EThOS for relevant terms: [TraVerse : a method of natural respiratory virus transmission from symptomatic children to healthy young adults](https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.755301) (`id: uk.bl.ethos.755301`)\n",
    "\n",
    "This command was used to create the initial folder structure:\n",
    "\n",
    "    ami-makeproject -p ethos --rawfiletypes pdf\n",
    "\n",
    "Resulting in a folder called `ethos` containing a JSON project file:\n",
    "\n",
    "```\n",
    "ethos/make_project.json\n",
    "```\n",
    "\n",
    "We then add the full text, as per the expected folder layout (called the [CProject](https://github.com/ContentMine/workshop-resources/blob/master/software-tutorials/cproject/README.md) naming conventions). Listing all files shows:\n",
    "\n",
    "```\n",
    "ethos/make_project.json\n",
    "ethos/uk.bl.ethos.755301/\n",
    "ethos/uk.bl.ethos.755301/fulltext.pdf\n",
    "```\n",
    "\n",
    "If you run the following code, and <i>if your browser can display PDF files</i>, the PDF should be displayed in the page below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame,display\n",
    "\n",
    "display(IFrame(src=\"ethos/uk.bl.ethos.755301/fulltext.pdf\", width=\"60%\", height=\"300px\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If that doesn't work, you can <a target=\"_blank\" href=\"ethos/uk.bl.ethos.755301/fulltext.pdf\">download it instead</a>.\n",
    "\n",
    "So we have the source document, but to perform the dictionary analysis we need either a plain text version in a file called:\n",
    "\n",
    "```\n",
    "ethos/uk.bl.ethos.755301/fulltext.pdf.txt\n",
    "```\n",
    "\n",
    "Or a HTML version in a file called:\n",
    "\n",
    "```\n",
    "ethos/uk.bl.ethos.755301/scholarly.html\n",
    "```\n",
    "\n",
    "The ContentMine toolkit provide wrappers for some sophisticated tools for performing PDF-to-HTML conversion, but for our purposes we can start with a simple HTML version generated by [Apache Tika](https://tika.apache.org/).\n",
    "\n",
    "The following sequence of shell commands can be used to generate a suitable HTML version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!echo \"Converting to HTML using Apache Tika...\"\n",
    "!java -jar /opt/tika.jar ethos/uk.bl.ethos.755301/fulltext.pdf > ethos/uk.bl.ethos.755301/scholarly.html\n",
    "!echo \"DONE!\"\n",
    "!ls -l ethos/uk.bl.ethos.755301/scholarly.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've re-run the script above, the simple HTML this process generated can be viewed here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"60%\"\n",
       "            height=\"300px\"\n",
       "            src=\"ethos/uk.bl.ethos.755301/scholarly.html\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7ff9ec49db00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import IFrame,display\n",
    "\n",
    "display(IFrame(src=\"ethos/uk.bl.ethos.755301/scholarly.html\", width=\"60%\", height=\"300px\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have HTML, we can finally extract some facts! We use the `ami-search` tool from the ContentMine [AMI3](https://github.com/petermr/ami3) project, which takes dictionaries of keywords relating to specific concepts and records where they appear in a set of texts.\n",
    "\n",
    "The arguments specify:\n",
    "\n",
    "- `-p ethos` to specify which project to process,\n",
    "- `--forcemake` to make sure the output is regenerated even if the output files already exist,\n",
    "- followed by a list of dictionaries to use, in this case the `country`, `virus_topics` and `virus_systemic_diseases` dictionaries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Generic values (AMISearchTool)\n",
      "================================\n",
      "-v to see generic values\n",
      "oldstyle            true\n",
      "\n",
      "Specific values (AMISearchTool)\n",
      "================================\n",
      "oldstyle             true\n",
      "strip numbers        false\n",
      "wordCountRange       (20,1000000)\n",
      "wordLengthRange      (1,20)\n",
      "\n",
      "dictionaryList       [country, virus_topics, viral_systemic_diseases]\n",
      "dictionaryTop        null\n",
      "dictionarySuffix     [xml]\n",
      "\n",
      "0    [main] DEBUG org.contentmine.ami.tools.AbstractAMISearchTool  - old style search command); change\n",
      "0 [main] DEBUG org.contentmine.ami.tools.AbstractAMISearchTool  - old style search command); change\n",
      "cProject: ethos\n",
      "legacy cmd> word(frequencies)xpath:@count>20~w.stopwords:pmcstop.txt_stopwords.txt\n",
      "legacy cmd> search(country)\n",
      "legacy cmd> search(virus_topics)\n",
      "legacy cmd> search(viral_systemic_diseases)\n",
      "uk.bl.ethos.755301 \n",
      "large document (7031) for uk.bl.ethos.755301 truncated to 500 sections\n",
      "...\n",
      "large document (7031) for uk.bl.ethos.755301 truncated to 500 sections\n",
      "...\n",
      "large document (7031) for uk.bl.ethos.755301 truncated to 500 sections\n",
      "...\n",
      "large document (7031) for uk.bl.ethos.755301 truncated to 500 sections\n",
      "...\n",
      "create data tables\n",
      "rrrr"
     ]
    }
   ],
   "source": [
    "!ami-search -p ethos --forcemake --dictionary=country --dictionary=virus_topics --dictionary=viral_systemic_diseases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look in the `ethos` folder we can see all the new files created by the `ami-search` process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ethos/search.viral_systemic_diseases.count.xml\r\n",
      "ethos/uk.bl.ethos.755301/search.viral_systemic_diseases.count.xml\r\n",
      "ethos/uk.bl.ethos.755301/word.frequencies.count.xml\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext.tei.xml\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext.tei.html\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-5.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-12.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-2.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-3.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-10.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-16.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-6.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-9.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-14.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-11.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-7.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-1.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-15.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-4.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-8.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-13.png\r\n",
      "ethos/uk.bl.ethos.755301/tei/fulltext_assets/image-17.png\r\n",
      "ethos/uk.bl.ethos.755301/search.virus_topics.snippets.xml\r\n",
      "ethos/uk.bl.ethos.755301/search.country.count.xml\r\n",
      "ethos/uk.bl.ethos.755301/scholarly.tika.html\r\n",
      "ethos/uk.bl.ethos.755301/scholarly.html\r\n",
      "ethos/uk.bl.ethos.755301/results/word/frequencies/results.html\r\n",
      "ethos/uk.bl.ethos.755301/results/word/frequencies/results.xml\r\n",
      "ethos/uk.bl.ethos.755301/results/search/country/results.xml\r\n",
      "ethos/uk.bl.ethos.755301/results/search/viral_systemic_diseases/results.xml\r\n",
      "ethos/uk.bl.ethos.755301/results/search/virus_topics/results.xml\r\n",
      "ethos/uk.bl.ethos.755301/word.frequencies.snippets.xml\r\n",
      "ethos/uk.bl.ethos.755301/fulltext.pdf\r\n",
      "ethos/uk.bl.ethos.755301/search.country.snippets.xml\r\n",
      "ethos/uk.bl.ethos.755301/search.virus_topics.count.xml\r\n",
      "ethos/uk.bl.ethos.755301/search.viral_systemic_diseases.snippets.xml\r\n",
      "ethos/word.frequencies.count.xml\r\n",
      "ethos/word.frequencies.documents.xml\r\n",
      "ethos/search.country.documents.xml\r\n",
      "ethos/full.dataTables.html\r\n",
      "ethos/search.virus_topics.snippets.xml\r\n",
      "ethos/search.country.count.xml\r\n",
      "ethos/search.viral_systemic_diseases.documents.xml\r\n",
      "ethos/search.virus_topics.documents.xml\r\n",
      "ethos/word.frequencies.snippets.xml\r\n",
      "ethos/search.country.snippets.xml\r\n",
      "ethos/__cooccurrence/country,virus_topics/histogram.svg\r\n",
      "ethos/__cooccurrence/country,virus_topics/histogram.csv\r\n",
      "ethos/__cooccurrence/virus_topics-virus_topics/cooccur.svg\r\n",
      "ethos/__cooccurrence/virus_topics-virus_topics/cooccur.csv\r\n",
      "ethos/__cooccurrence/country/histogram.svg\r\n",
      "ethos/__cooccurrence/country/histogram.csv\r\n",
      "ethos/__cooccurrence/country-country/cooccur.svg\r\n",
      "ethos/__cooccurrence/country-country/cooccur.csv\r\n",
      "ethos/__cooccurrence/viral_systemic_diseases-viral_systemic_diseases/cooccur.svg\r\n",
      "ethos/__cooccurrence/viral_systemic_diseases-viral_systemic_diseases/cooccur.csv\r\n",
      "ethos/__cooccurrence/country-virus_topics/cooccur.svg\r\n",
      "ethos/__cooccurrence/country-virus_topics/cooccur.csv\r\n",
      "ethos/__cooccurrence/country-viral_systemic_diseases/cooccur.svg\r\n",
      "ethos/__cooccurrence/country-viral_systemic_diseases/cooccur.csv\r\n",
      "ethos/__cooccurrence/viral_systemic_diseases/histogram.svg\r\n",
      "ethos/__cooccurrence/viral_systemic_diseases/histogram.csv\r\n",
      "ethos/__cooccurrence/allPlots.svg\r\n",
      "ethos/__cooccurrence/virus_topics-viral_systemic_diseases/cooccur.svg\r\n",
      "ethos/__cooccurrence/virus_topics-viral_systemic_diseases/cooccur.csv\r\n",
      "ethos/__cooccurrence/virus_topics/histogram.svg\r\n",
      "ethos/__cooccurrence/virus_topics/histogram.csv\r\n",
      "ethos/__cooccurrence/country,virus_topics-country,virus_topics/cooccur.csv\r\n",
      "ethos/commonest.dataTables.html\r\n",
      "ethos/count.dataTables.html\r\n",
      "ethos/entries.dataTables.html\r\n",
      "ethos/make_project.json\r\n",
      "ethos/search.virus_topics.count.xml\r\n",
      "ethos/search.viral_systemic_diseases.snippets.xml\r\n"
     ]
    }
   ],
   "source": [
    "!find ethos -type f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"300px\"\n",
       "            src=\"ethos/full.dataTables.html\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f2ed8318f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(IFrame(src=\"ethos/full.dataTables.html\", width=\"100%\", height=\"300px\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_TBA look at the results..._ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"300px\"\n",
       "            src=\"ethos/uk.bl.ethos.755301/results/search/virus_topics/results.xml\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f2ed8227160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(IFrame(src=\"ethos/uk.bl.ethos.755301/results/search/virus_topics/results.xml\", width=\"100%\", height=\"300px\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"50%\"\n",
       "            height=\"350px\"\n",
       "            src=\"ethos/__cooccurrence/country-virus_topics/cooccur.svg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f2ed037d278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import SVG,display\n",
    "#display(SVG(filename='ethos/__cooccurrence/viral_systemic_diseases-viral_systemic_diseases/cooccur.svg'))\n",
    "#display(SVG(filename='ethos/__cooccurrence/allPlots.svg', height=\"100px\"))\n",
    "display(IFrame(src=\"ethos/__cooccurrence/country-virus_topics/cooccur.svg\", width=\"50%\", height=\"350px\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've only processed a single document, so the co-occurance matrix for the terms that appear in the document is `1` everywhere!  To get something more interesting, we'll need to analyse more documents.\n",
    "\n",
    "# Conclusion\n",
    "\n",
    "Hopefully, this notebook has introduced the major concepts and illustrated the potential uses of the ContentMine approach and tools. \n",
    "\n",
    "_TBA next steps_\n",
    "\n",
    "* Process many theses.\n",
    "* [other notebooks](/) on GROBID, OCR, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}