cwpk-64-embedding-entity-recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#64: Embeddings, Summarization and Entity Recognition\n",
    "=======================================\n",
    "\n",
    "Some Machine Learning Applied to NLP Problems\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In the last installment of [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) we collected roughly 45,000 articles from Wikipedia that match KBpedia reference concepts. We then did some pre-processing of the text using the [gensim](https://radimrehurek.com/gensim/) package to lower case the text, remove stop words, and identify bi-gram and tri-gram phrases. These types of functions and extractions are a strength of gensim, which should be part of your pre-processing arsenal.\n",
    "\n",
    "It is now time for us to process the natural language of this text and to use it for creating word and document embedding models. For the later, we will continue to use gensim. For the former, however, we will introduce and use a very powerful NLP package, [spaCy](https://spacy.io/). As noted earlier, spaCy has a clear function orientation to NLP and also has some powerful extension mechanisms.\n",
    "\n",
    "Our plan of attack in this installment is to finish the word embeddings with gensim, and then move on to explore the spaCy package. We will not explore all aspects of this package, but will focus on [text summarization](https://en.wikipedia.org/wiki/Automatic_summarization), and [(named) entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition) using both models and rule-based."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word and Document Embedding\n",
    "As we have noted in [**CWPK #61**](https://www.mkbergman.com/2411/cwpk-61-nlp-machine-learning-and-analysis/), there exist many pre-calculated word and document embedding models. However, because of the clean scope of KBpedia and our interest in manipulating embeddings for various learning models, we want to create our own embeddings.\n",
    "\n",
    "There are many scopes and methods for creating embeddings. Embeddings, you recall, are a vector representation of information in a reduced dimensional space. Embeddings are a proven way to represent sparse matrix information like language (meaning many dimensions of words and phrases matched to one another) in a more efficient coding format usable by a computer. Embedding scopes may range from words, phrases, sentences, paragraphs, sections of documents, or [documents](https://radimrehurek.com/gensim/models/doc2vec.html), as well as [senses](https://github.com/explosion/sense2vec), [topics](https://github.com/askerlee/topicvec), sentiments, [categories](https://github.com/rakuten-nlp/category2vec) or other relations that might cut across a given corpus. Methods may range from sequences to counts to simple statistics or all the way up to deep learning with neural nets. Of late, a combination method of converging encoders and decoders called 'transformers' has been the rage, with [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) and [ELMo](https://en.wikipedia.org/wiki/ELMo) two prominent instantiations.\n",
    "\n",
    "Because we already have been exercising the gensim package, we decide to proceed with our own [word embedding](https://stackabuse.com/implementing-word2vec-with-gensim-library-in-python/) and [document embedding](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-wikipedia.ipynb) models. From gensim documentation, we first prepare up a [word2vec](https://radimrehurek.com/gensim/models/word2vec.html) model:\n",
    "\n",
    "<div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"><span style=\"font-weight: bold;\">NOTE:</span> Due to GitHub's file size limits, the various text file inputs referenced in this installment may be found on the KBpedia site as zipped files (for example, <a href=\"https://kbpedia.org/cwpk-text/wikipedia-trigram.zip\">https://kbpedia.org/cwpk-text/wikipedia-trigram.zip</a> for the input file mentioned next). Due to their very large sizes, you will need to create locally all of the models mentioned in this installment (with <code>*.vec</code> or <code>*.model</code> extensions).</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models.word2vec import LineSentence\n",
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-trigram.txt'\n",
    "out_model = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-w2v.model'\n",
    "out_vec = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-w2v.vec'\n",
    "\n",
    "input = smart_open(in_f, 'r', encoding='utf-8')\n",
    "walks = LineSentence(input)\n",
    "model = Word2Vec(walks, size=300, window=5, min_count=3, hs=1, sg=1, workers=5, negative=5, iter=10)\n",
    "model.save(out_model)\n",
    "model.wv.save_word2vec_format(out_vec, binary=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works pretty slick and only requires a few lines of code. The model run takes about 2:15 hrs on my laptop; to process the entire Wikipedia English corpus reportedly takes about 9 hrs. Note we begin the training process with our tri-gram input corpus created in the last installment.\n",
    "\n",
    "A few of the model parameters deserve some commentary. The <code>size</code> parameter is one of the most important. It sets the number of dimensions over which you want to capture a correspondence statistic, what is the actual [dimension reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction) at the core of the entire exercise. Remember, a collocation matrix is a very sparse one for natural language. In the case of how I have set up the Wikipedia pages from KBpedia so far with stoplist and trigrams and such, our current corpus has 1.3 million tokens, which is really sparse when you extend the second dimension by this same amount. The <code>size</code> parameter beyond hundreds of dimensions works to greatly increase the computation time in training as well as (perhaps paradoxically) lowering accuracy. The <code>window</code> parameter is the word count to either side of the current token for which adjacency is calculated, so that a <code>window</code> of five actually encompasses a string of eleven tokens, the subject token and five to either side. <code>min-counts</code> is the minimum number of occurrences for a given token (including phrases or ngrams as individual tokens). <code>sg</code> in this case is invoking the 'skip-gram' method as opposed to the second method more commonly used, the 'cbow' (continuous bag of words) method. \n",
    "\n",
    "Like any central Python function, you should study this one to learn more about some of the other settable parameters. What is most important, however, is to learn about these settings, test those you deem critical, and realize fine-tuning such parameters is likely the key to successful results with your machine learning efforts. It is a common secret that success with machine learning is dependent on setting up and then tweaking the parameters that go into any particular method. \n",
    "\n",
    "We can take this same code block above and set up the [doc2vec](https://radimrehurek.com/gensim/models/doc2vec.html) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from gensim.models.doc2vec import Doc2Vec, TaggedLineDocument\n",
    "from gensim.models import Word2Vec\n",
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-trigram.txt'\n",
    "out_model = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-d2v.model'\n",
    "out_vec = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-d2v.vec'\n",
    "input = smart_open(in_f, 'r', encoding='utf-8')\n",
    "\n",
    "documents = TaggedLineDocument(input)\n",
    "training_src = list(documents)\n",
    "print(training_src[:1])\n",
    "model = Doc2Vec(vector_size=300, min_count=15, epochs=30)\n",
    "model.build_vocab(training_src)\n",
    "model.train(training_src, total_examples=model.corpus_count, epochs=model.epochs)\n",
    "model.save(out_model)\n",
    "model.save_word2vec_format(out_vec, binary=False)\n",
    "print(model.infer_vector(['violent', 'means', 'to', 'destroy', 'the','organization']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The doc2vec method has a similar setup. The main difference is that the vector calculation is now based on full sentences versus individual words. We also increase the <code>min_count</code> parameter. We'll see the results of this training in the next section.\n",
    "\n",
    "gensim also has methods to train [FastText](https://radimrehurek.com/gensim/models/fasttext.html). Please consult the documentation for this method as well as to understand better the various training parameters. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity Analysis\n",
    "A good way to see the effect of embedding vectors is through similarity analysis. The calculations are based on the adjacency of vectors in the embedding space.\n",
    "\n",
    "Our first two examples use word2vec for our newly created KBpedia-Wikipedia corpus. The first example calculates the relatedness between two entered terms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6882485"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "path = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-w2v.model'\n",
    "model = Word2Vec.load(path)\n",
    "model.wv.similarity('man', 'woman')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second example retrieves the most closely related terms given an input term or phrase (in this case, <code>machine_learning</code>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('artificial_intelligence', 0.7738381624221802),\n",
       " ('data_mining', 0.7659739255905151),\n",
       " ('algorithms', 0.7430499792098999),\n",
       " ('natural_language_processing', 0.7429415583610535),\n",
       " ('computational', 0.7116029262542725),\n",
       " ('computational_linguistics', 0.6903550028800964)]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "path = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-w2v.model'\n",
    "model = Word2Vec.load(path)\n",
    "w1 = ['machine_learning']\n",
    "model.wv.most_similar(positive=w1, topn=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "gensim offers a number of settings including whether one can analyze without training (effectively a 'read only' option) and other parameters including number of results returned, etc.\n",
    "\n",
    "We can also compare the doc2vec approach in comparison to word2vec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('quantitative_methods', 0.4042782783508301),\n",
       " ('artificial_intelligence', 0.3983246088027954),\n",
       " ('evolutionary_computation', 0.39264559745788574),\n",
       " ('information_retrieval', 0.38776731491088867),\n",
       " ('natural_language_processing', 0.38531848788261414),\n",
       " ('deep_learning', 0.37803560495376587)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import Doc2Vec\n",
    "\n",
    "path = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\wikipedia-d2v.model'\n",
    "model = Doc2Vec.load(path)\n",
    "w1 = ['machine_learning']\n",
    "model.wv.most_similar(positive=w1, topn=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note we get a similar listing of results, though the correlation scores in this doc2vec case are much lower.\n",
    "\n",
    "These efforts conclude our embedding tests for the moment. We will be adding additional embeddings based on knowledge graph **structure** and **annotations** in **CWPK #67**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Summarization\n",
    "Let's now switch gears and introduce our basic natural language processing package, <code>spaCy</code>. Out-of-the-box <code>spaCy</code> includes the standard [NLP utilities](https://spacy.io/usage/linguistic-features) of part-of-speech tagging, lemmatization, dependency parsing, named entity recognition, entity linking, tokenization, merging and splitting, and sentence segmentation. Various vector embedding or rule-based processing methods may be layered on top of these utilities, and they may be combined into flexible NLP processing pipelines. \n",
    "\n",
    "We are not doing anything special here, but I wanted to include text summarization because it nicely combines many functions and utilities provided by the <code>spaCy</code> package. Here is an example using an existing <code>spaCy</code> model, <code>en_core_web_sm</code>, which has pre-calculated POS and NER tags based on the English [OntoNotes 5](https://catalog.ldc.upenn.edu/LDC2013T19) corpus. (You will need to separately download and install these existing models.) The text to be evaluated was copied-and-pasted from [**CWPK #61**](https://www.mkbergman.com/2411/cwpk-61-nlp-machine-learning-and-analysis/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "from string import punctuation\n",
    "import en_core_web_sm\n",
    "\n",
    "extra_words = list(STOP_WORDS) + list(punctuation) + ['\\n']\n",
    "nlp = en_core_web_sm.load()\n",
    "doc = \"\"\"With this installment of the Cooking with Python and KBpedia series we move into Part VI of seven parts, a part with the bulk of the analytical and machine learning (that is, \"data science\") discussion, and the last part where significant code is developed and documented. At the conclusion of this part, which itself has 11 installments, we have four installments to wrap up the series and provide a consistent roadmap to the entire project.\n",
    "      Knowledge graphs are unique information artifacts, and KBpedia is further unique in terms of its consistent and logical construction as well as its incorporation of significant text content via Wikipedia pages. These characteristics provide unique value for KBpedia, but it is also a combination not duplicated anywhere else in the data science ecosystem. One of the objectives, therefore, of this part of our CWPK series is the creation of some baseline knowledge representations useful to data science aims that capture these unique characteristics.\n",
    "      KBpedia's (or any knowledge graph constructed in a similar manner) combination of characteristics make it a powerful resource in three areas of data science and machine learning. First, the nearly universal scope and degree of topic coverage with about 56,000 concepts, logically organized into typologies with a high degree of disjointedness, means that accurate 'slices' or training sets may be extracted from KBpedia nearly instantaneously. Labeled training sets are one of the most time consuming and expensive activities in doing supervised machine learning. We can extract these nearly for free from KBpedia. Further, with its links to tens of millions of entities in its mapped knowledge bases such as Wikidata, literally tens of thousands of conceptual entities in KBpedia can be the retrieval points to nucleate training sets for fine-grained entity recognition.\n",
    "      Second, 80% of KBpedia's concepts are mapped to Wikipedia articles. While many Wikipedia-based word embedding models exist, the ones in KBpedia are logically categorized and have rough equivalence in terms of scope and prominence, hopefully providing cleaner topic 'signals'. To probe these assertions, we will create a unique KBpedia-based word embedding corpus that also leverages labels for items of structural importance, such as typology membership. We will use this corpus in many of our tests and as a general focus in our training sets.\n",
    "      And, third, perhaps the most important area, knowledge graphs offer unique structures and challenges for machine learning, especially innovations in geometric, heterogeneous methods for deep learning. The first generation of deep machine learning was designed for grid-patterned data and matrices through approaches such as deep neural networks, convolutional neural networks (CNN ), or recurrent neural networks (RNN). The 'deep' appelation comes from having multiple calculated, intermediate layers of transformations between the grid inputs and outputs for the model. Graphs, on the other hand, are heterogeneous between nodes and edges. They may be directed (subsumptive) in nature. And, for knowledge graphs, they have much labeling and annotation, including varying degrees of attribute completeness. Language embedding, itself often a product of deep learning, enables the efficient incorporation of text. It is only in the past five years that concerted attention has been devoted to better capturing this feature richness for knowledge graphs.\n",
    "      The eleven installments in this part will look in more depth at networks and graphs, focus on how to create training sets and embeddings for the learners, discuss some natural language packages and uses, and then look in depth at 'standard' machine learners and deep learners. We will install the first generation of deep graph learners and then explore some on the cutting edge. We will test many use cases, but will also try to invoke classifiers across this spectrum so that we can draw some general conclusions.\n",
    "      The material below introduces and tees up these topics. We describe leading Python packages for data science, and how we have architected our own approach, We have picked a particular Python machine learning framework, PyTorch, to which we will then tie four different NLP and deep learning libraries. We devote two installments each to these four libraries. The use cases we document across these installments are in addition to the existing ones we have in Clojure posted online.\n",
    "      So, we think we have an interesting suite of benefits to cover in this part, some arising from being based on KBpedia and some arising from the nature of knowledge graphs. On the other hand, due to the relative immaturity of the field, we are still actively learning and innovating around the juncture of AI and knowledge graphs. Thus, one of the reasons we emphasize Python 'ecosystems' and 'frameworks' in this part is to be better prepared to incorporate those innovations and learnings to come.\n",
    "      One of the first prototypes of machine learning comes from the statistician Ronald Fisher in the 1930s regarding how to classify Iris species based on the attributes of their flowers. It was a multivariate data example using the method we today call linear discriminant analysis. This classic example is still taught. But many dozens of new algorithms and combined approaches have joined the machine learning field since then.\n",
    "      Figure 1 below is one way to characterize the field, with ML standing for machine learning and DL for deep learning, with this one oriented to sub-fields in which some Python package already exists:\n",
    "      There are many possible diagrams that one might prepare to show the machine learning landscape, including ones with a larger emphasis on text and knowledge graphs. Most all schematics of the field show a basic split between supervised learning and unsupervised learning, (sometimes with reinforcement learning as another main branch), with the main difference being that supervised approaches iterate to achieve statistical fit with pre-determined labels, whereas unsupervised is unlabeled. Accurate labeling can be costly and time consuming. Note that the idea of 'classification' is a supervised one, 'clustering' a notion of unsupervised.\n",
    "      We will include a 'standard' machine learning library in our proposed toolkit, the selection of which I discuss below. However, the most evaluation time I spent in researching these installments was directed to the idea of knowledge representation and embeddings applicable to graphs. Graphs pose a number of differences and challenges to standard machine learning. They have only been a recent (5 yr) focus in machine learning, which is also rapidly changing over time.\n",
    "      All machine learners need to operate on their feature spaces in numerical representations. Text is a tricky form because language is difficult and complex, and how to represent the tokens within our language usable by a computer needs to consider, what? Parts-of-speech, the word itself, sentence construction, semantic meaning, context, adjacency, entity recognition or characterization? These may all figure into how one might represent text. Machine learning has brought us unsupervised methods for converting words to sentences to documents and, now, graphs, to a reduced, numeric representation known as \"embeddings.\" The embedding method may capture one or more of these textual or structural aspects.\n",
    "      Much of the first interest in machine learning based on graphs was driven by these interests in embeddings for language text. Standard machine classifiers with deep learning using neural networks have given us word2vec, and more recently BERT and its dozens of variants have reinforced the usefulness of deep learning to create pre-trained text representations.\n",
    "      Indeed, embeddings do figure prominently in knowledge graph representation, but only as one among many useful features. Knowledge graphs with hierarchical (subsumption) relationships, as might be found in any taxonomy, become directed. Knowledge graphs are asymmetrical, and often multi-typed and sometimes multi-modal. There is heterogeneity among nodes and links or edges. Not all knowledge graphs are created equal and some of these aspects may not apply. Whether there is an accompanying richness of text description that accompanies the node or edges is another wrinkle. None of the early CNN or RNN or simple neural net approaches match well with these structures.\n",
    "      The general category that appears to have emerged for this scope is geometric deep learning, which applies to all forms of graphs and manifolds. There are other nuances in this area, for example whether a static representation is the basis for analysis or one that is dynamic, essentially allowing learning parameters to be changed as the deep learning progresses through its layers. But GDL has the theoretical potential to address and incorporate all of the wrinkles associated with heterogeneous knowledge graphs.\n",
    "      So, this discussion helps define our desired scope. We want to be able to embrace Python packages that range from simple statistics to simple machine learning, throwing in natural language processing and creating embedding representations, that can then range all the way through deep learning to the cutting-edge aspects of geometric or graph deep learning.\n",
    "      This background provides the necessary context for our investigations of Python packages, frameworks, or libraries that may fulfill the data science objectives of this part. Our new components often build upon and need to play nice with some of the other requisite packages introduced in earlier installments, including pandas ([CWPK #55]), NetworkX ([CWPK #56]), and PyViz ([CWPK #55]). NumPy has been installed, but not discussed.\n",
    "      It is not fair to say that natural language processing has become a 'commodity' in the data science space, but it is also true there is a wealth of capable, complete packages within Python. There are standard NLP requirements like text cleaning, tokenization, parts-of-speech identification, parsing, lemmatization, phrase identification, and so forth. We want these general text processing capabilities since they are often building blocks and sometimes needed in their own right. We also would like to add to this baseline such considerations as interoperability, creating embeddings, or other special functions.\n",
    "      Another key area is language embedding. Language embeddings are means to translate language into a numerical representation for use in downstream analysis, with great variety in what aspects of language are captured and how to craft them. The simplest and still widely-used representation is tf-idf (term frequency–inverse document frequency) statistical measure. A common variant after that was the vector space model. We also have latent (unsupervised) models such as LDA. A more easily calculated option is explicit semantic analysis (ESA). At the word level, two of the prominent options are word2vec and gloVe, which is used directly in spaCy. These have arisen from deep learning models. We also have similar approaches to represent topics (topicvec), sentences (sentence2vec), categories and paragraphs (Category2Vec), documents (doc2vec), node2vec or entire languages (BERT and variants and GPT-3 and related methods). In all of these cases, the embedding consists of reducing the dimensionality of the input text, which is then represented in numeric form.\n",
    "      There are internal methods for creating embeddings in multiple machine learning libraries. Some packages are more dedicated, such as fastText, which is a library for learning of word embeddings and text classification created by Facebook's AI Research (FAIR) lab. Another option is TextBrewer, which is an open-source knowledge distillation toolkit based on PyTorch and which uses (among others) BERT to provide text classification, reading comprehension, NER or sequence labeling.\n",
    "      Closely related to how we represent text are corpora and datasets that may be used either for reference or training purposes. These need to be assembled and tested as well as software packages. The availability of corpora to different packages is a useful evaluation criterion. But, the picking of specific corpora depends on the ultimate Python packages used and the task at hand. We will return to this topic in CWPK #63.\n",
    "      Of course, nearly all of the Python packages mentioned in this Part VI have some relation to machine learning in one form or another. I call out this category separately because, like for NLP, I think it makes sense to have a general machine learning library not devoted to deep learning but providing a repository of classic learning methods.\n",
    "      There really is no general option that compares with scikit-learn. It features various classification, regression, and clustering algorithms, including support vector machines, random forests, gradient boosting, k-means and DBSCAN data clustering, and is designed to interoperate with NumPy and SciPy. The project is extremely active with good documentation and examples.\n",
    "      Deep learning is characterized by many options, methods and philosophies, all in a fast-changing area of knowledge. New methods need to be compared on numerous grounds from feature and training set selection to testing, parameter tuning, and performance comparisons. These realities have put a premium on libraries and frameworks that wrap methods in repeatable interfaces and provide abstract functions for setting up and managing various deep (and other) learning algorithms.\n",
    "      The space of deep learning thus embraces many individual methods and forms, often expressed through a governing ecosystem of other tools and packages. These demands lead to a confusing and overlapping and non-intersecting space of Python options that are hard to describe and comparatively evaluate. Here are some of the libraries and packages that fit within the deep and machine learning space, including abstraction frameworks:\n",
    "      Theano is a Python library and optimizing compiler for manipulating and evaluating mathematical expressions, especially matrix-valued ones; it is tightly integrated with NumPy, and uses it at the lowest level.\n",
    "      Keras is increasingly aligning with TensorFlow and some, like Chainer and CNTK, are being deprecated in favor of the two leading gorillas, PyTorch and TensorFlow. One approach to improve interoperability is the Open Neural Network Exchange (ONNX) with the repository available on GitHub. There are existing converters to ONNX for Keras, TensorFlow, PyTorch and scikit-learn.\n",
    "      A key development from deep learning of the past three years has been the usefulness of Transformers, a technique that marries encoders and decoders converging on the same representation. The technique is particularly helpful to sequential data and NLP, with state-of-the-art performance to date for: next-sentence prediction, question answering, reading comprehension, sentiment analysis, and paraphrasing.\n",
    "      Both BERT and GPT are pre-trained products that utilize this method. Both TensorFlow and PyTorch contain Transformer capabilities.\n",
    "      As noted, most of my research for this Part VI has resided in the area of a subset of deep graph learning applicable to knowledge graphs. The leading deep learning libraries do not, in general, provide support for this area of representational learning, sometimes called knowledge representation learning (KRL) or knowledge graph embedding (KGE). Within this rather limited scope, most options also seem oriented to link prediction and knowledge graph completion (KGC), rather than the heterogeneous aspects with text and OWL2 orientation characteristic of KBpedia.\n",
    "      We thus see this rough hierachy: machine learning → deep learning → geometric deep learning → graph (R) learning → KG learning\n",
    "      Lastly, more broadly, there is the recently announced KGTK, which is a generalized toolkit with broader capabilities for large scale knowledge graph manipulation and analysis. KGTK also puts forward a standard KG file format, among other tools.\n",
    "      A Generalized Python Data Science Architecture\n",
    "      With what we already have in hand, plus the libraries and packages described above, we have a pretty good inventory of candidates to choose from in proceeding with our next installments. Like our investigations around graphics and visualization (see [CWPK #55]), the broad areas of data science, machine learning, and deep learning have been evolving to one of comprehensive ecosystems. Figure 2 below presents a representation of the Python components that make sense for the machine learning and application environment. As noted, our local Windows machines lack separate GPUs (graphical processing units), so the hardware is based on a standard CPU (which has an integrated GPU that can not be separately targeted). We have already introduced and discusses some of the major Python packages and libraries, including pandas, NetworkX, and PyViz. Here is that representative data science architecture:\n",
    "      Representative Python Components\n",
    "      The defining architectural question for this Part VI is what general deep and machine learning framework we want (if any). I think using a framework makes sense over scripting together individual packages, though for some tests that still might be necessary. If I was to adopt a framework, I would also want one that has a broad set of tools in its ecosystem and common and simpler ways to define projects and manage the overall pipelines from data to results. As noted, the two major candidates appear to be TensorFlow and PyTorch.\n",
    "      TensorFlow has been around the longest, has, today, the strongest ecosystem, and reportedly is better for commercial deployments. Google, besides being the sponsor, uses TensorFlow in most of its ML projects and has shown a commitment to compete with the upstart PyTorch by significantly re-designing and enhancing TensorFlow 2.0.\n",
    "      On the other hand, I very much like the more 'application' orientation of PyTorch. Innovation has been fast and market share has been rising.\n",
    "      Though some of the intriguing packages for TensorFlow are not apparently available for PyTorch, including Graph Nets, Keras, Plaid ML, and StellarGraph, PyTorch does have these other packages not yet mentioned that look potentially valuable down the road:\n",
    "      One disappointment is that neither of these two leading packages directly ingest [RDFLib] graph files, though with PyTorch and DGL you can import or export a NetworkX graph directly. pandas is also a possible data exchange format.\n",
    "      Consideration of all of these points has led us to select PyTorch as the initial data science framework. It is good to know, however, that a fairly comparable alternative also exists with TensorFlow and Keras.\n",
    "      Finally, with respect to Figure 2 above, we have no plans at present to use the Dask package for parallelizing analytic calculations.\n",
    "      With the PyTorch decision made, at least for the present, we are now clear to deal with specific additional packages and libraries. I highlight four of these in this section. Each of these four is the focus of two separate installments as we work to complete this Part VI. One of these four is in natural language processing (spaCy), one in general machine learning (scikit-learn), and two in deep learning with an emphasis on graphs (DGL and DGL-KE, and PyG). These choices again tend to reinforce the idea of evaluating whole ecosystems, as opposed to single packages. Note, of course, that more specifics on these four packages will be presented in the forthcoming installments.\n",
    "      I find spaCy to be very impressive, with many potentially useful extensions including sense2vec, spacy-stanza, spacy-wordnet, torchtext, and GenSim.\n",
    "      The major competitor is NLTK. The reputation of this package is stellar and it has proven itself for decades. It is a more disaggregate approach by scholars and researchers to enable users to build complex NLP functionality. It is therefore harder to use and configure, and is also less performant. The real differentiator, however, is the more object or application orientation of spaCy.\n",
    "      Though NLTK appears to have good NLP tools for processing data pipelines using text, most of these functions appear to be in spaCy and there are also the Flair and PyTorch-NLP packages available in the PyTorch environment if needed. GenSim looks to be a very useful enhancement to the environment because of the advanced text evaluation modes it offers, including sentiment. Not all of these will be tested during this CWPK series, but it will be good to have these general capabilities resident in cowpoke.\n",
    "      We earlier signaled our intent to embrace scikit-learn, principally to provide basic machine learning support. scikit-learn provides a unified API to these basic tasks, including crafting pipelines and meta-functions to integrate the data flow. scikit-learn works on any numeric data stored as NumPy arrays or SciPy sparse matrices. Other types that are convertible to numeric arrays such as pandas DataFrame are also acceptable.\n",
    "      Some of the general ML methods, and there are about 40 supervised ones in the package, may be useful and applicable to specific circumstances include: dimensionality reduction, model testing, preprocessing, scoring methods, and principal component analysis (PCA).\n",
    "      A real test of this package will be ease of creating (and then being able to easily modify) data processing and analysis pipelines. Another test will be ingesting, using, and exporting data formats useful to the KBpedia knowledge graph. We know that scikit-learn doesn't talk directly to NetworkX, though there may be recipes for the transfer; graphs are represented in scikit-learn as connectivity matrices. pandas can interface via common formats including CSV, Excel, JSON and SQL, or, with some processing, DataFrames. scikit-learn supports data formats from NumPy and SciPy, and it supports a datasets.load_files format that may be suitable for transferring many and longer text fields. One option that is intriguing is how to leverage the CSV flat-file orientation of our KG build and extract routines in cowpoke for data transfer and transformation.\n",
    "      I also want to keep an eye on the possible use of skorch to better integrate with the overall PyTorch environment, or to add perhaps needed and missing functionality or ease of development. There is much to explore with these various packages and environments.\n",
    "      For our basic, 'vanilla', deep graph analysis package we have chosen the eponymous Deep Graph Library for basic graph neural network operations, which may run on CPU or GPU machines or clusters. The better interface relevant to KBpedia is through DGL-KE, a high performance, reportedly easy-to-use, and scalable package for learning large-scale knowledge graph embeddings that extends DGL. DGL-KE also comes configured with the popular models of TransE, TransR, RESCAL, DistMult, ComplEx, and RotatE.\n",
    "      PyTorch Geometric is closely tied to PyTorch, and most impressively has uniform wrappers to about 40 state-of-art graph neural net methods. The idea of \"message passing\" in the approach means that heterogeneous features such as structure and text may be combined and made dynamic in their interactions with one another. Many of these intrigued me on paper, and now it will be exciting to test and have the capability to inspect these new methods as they arise. DeepSNAP may provide a direct bridge between NetworkX and PyTorch Geometric.\n",
    "      During the research on this Part VI I encountered a few leads that are either not ready for prime time or are off scope to the present CWPK series. A potentially powerful, but experimental approach that makes sense is to use SPARQL as the request-and-retrieval mechanism against the graph to feed the machine learners. RDFFrames provides an imperative Python API that gets internally translated to SPARQL, and it is integrated with the PyData machine learning software stack; see GitHub. Some methods above also use SPARQL. One of the benefits of a SPARQL approach, besides its sheer query and inferencing power, is the ability to keep the knowledge graph intact without data transform pipelines. It is quite available to serve up results in very flexible formats. The relative immaturity of the approach and performance considerations may be difficult challenges to overcome.\n",
    "      I earlier mentioned KarateClub, a Python framework combining about 40 state-of-the-art unsupervised graph mining algorithms in the areas of node embedding, whole-graph embedding, and community detection. It builds on the packages of NetworkX, PyGSP, Gensim, NumPy, and SciPy. Unfortunately, the package does not support directed graphs, though plans to do so have been stated. This project is worth monitoring.\n",
    "      A third intriguing area involves the use of quaternions based on Clifford algebras in their machine learning codes. Charles Peirce, the intellectual guide for the design of KBpedia, was a mathematician of some renown in his own time, and studied and applauded William Kingdon Clifford and his emerging algebra as a contemporary in the 1870s, shortly before Clifford's untimely death. Peirce scholars have often pointed to this influence in the development of Peirce's own algebras. I am personally interested in probing this approach to learn a bit more of Peirce's manifest interests.\n",
    "      \"\"\"\n",
    "docx = nlp(doc)\n",
    "all_words = [word.text for word in docx]\n",
    "Freq_word = {}\n",
    "for w in all_words:\n",
    "    w1 = w.lower()\n",
    "    if w1 not in extra_words and w1.isalpha():\n",
    "        if w1 in Freq_word.keys():\n",
    "              Freq_word[w1] += 1\n",
    "        else:\n",
    "              Freq_word[w1] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These <code>spaCy</code> models have to be separately loaded. In our case, we first needed to download and install the model package:\n",
    "\n",
    "Then, we needed to <code>import</code> the model as shown. To use a different model one would need to import and load that model separately.\n",
    "\n",
    "Let's go ahead and run this routine to get the word frequencies in the input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Freq_word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get an estimate of the overall topic for our input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic of document given :-\n",
      "machine learning deep "
     ]
    }
   ],
   "source": [
    "val=sorted(Freq_word.values())\n",
    "max_freq = val[-3:]\n",
    "print('Topic of document given :-')\n",
    "for word,freq in Freq_word.items():\n",
    "    if freq in max_freq:\n",
    "        print(word ,end = ' ')\n",
    "    else:\n",
    "        continue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now proceed to begin our text summarization by scoring the relevance of the sentences in the input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in Freq_word.keys():\n",
    "    Freq_word[word] = (Freq_word[word] / max_freq[-1])\n",
    "sent_strength = {}\n",
    "for sent in docx.sents:\n",
    "    sen_len = len(sent)\n",
    "    if sen_len >= 8:\n",
    "        for word in sent :\n",
    "            if word.text.lower() in Freq_word.keys():\n",
    "                if sent in sent_strength.keys():\n",
    "                    sent_strength[sent] += Freq_word[word.text.lower()]\n",
    "                else:\n",
    "                    sent_strength[sent] = Freq_word[word.text.lower()]\n",
    "            else: \n",
    "                continue\n",
    "    else:\n",
    "        sent_strength[sent] = 0\n",
    "top_sentences = (sorted(sent_strength.values())[::-1])\n",
    "top5percent_sentence = int(0.05 * len(top_sentences))\n",
    "top_sent = top_sentences[:top5percent_sentence]                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then, based on those scores, to generate a summary based on the top 5 percent of sentences in the input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And, third, perhaps the most important area, knowledge graphs offer unique structures and challenges for machine learning, especially innovations in geometric, heterogeneous methods for deep learning.We describe leading Python packages for data science, and how we have architected our own approach, We have picked a particular Python machine learning framework, PyTorch, to which we will then tie four different NLP and deep learning libraries.Standard machine classifiers with deep learning using neural networks have given us word2vec, and more recently BERT and its dozens of variants have reinforced the usefulness of deep learning to create pre-trained text representations.\n",
      "      We want to be able to embrace Python packages that range from simple statistics to simple machine learning, throwing in natural language processing and creating embedding representations, that can then range all the way through deep learning to the cutting-edge aspects of geometric or graph deep learning.\n",
      "      I call out this category separately because, like for NLP, I think it makes sense to have a general machine learning library not devoted to deep learning but providing a repository of classic learning methods.\n",
      "      The leading deep learning libraries do not, in general, provide support for this area of representational learning, sometimes called knowledge representation learning (KRL) or knowledge graph embedding (KGE).→ deep learning → geometric deep learning → graph (R) learningLike our investigations around graphics and visualization (see [CWPK #55]), the broad areas of data science, machine learning, and deep learning have been evolving to one of comprehensive ecosystems.One of these four is in natural language processing (spaCy), one in general machine learning (scikit-learn), and two in deep learning with an emphasis on graphs (DGL and DGL-KE, and PyG)."
     ]
    }
   ],
   "source": [
    "summary = []\n",
    "for sent,strength in sent_strength.items():\n",
    "    if strength in top_sent:\n",
    "        summary.append(sent)\n",
    "    else:\n",
    "        continue\n",
    "for i in summary:\n",
    "    print(i, end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can vary the size of the summarization by varying the percentage of sentences to be included. Note there are other methods available for summarizing text using <code>spaCy</code>. A standard search will turn up other methods than simply calculating top sentences based on word frequencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Named) Entity Recognition - Update Model\n",
    "Depending on the input model, <code>spaCy</code> provides pre-trained entity recognition models (the area is most often refered to as NER, but an actual category of entities need not be named with capitalization, which is why I prefer 'entity recognition'). In the case of the pre-trained English OntoNotes 5 model, these entity tags are:\n",
    "\n",
    "\n",
    "|  Type  |  Description  |\n",
    "|:--------:|:---------------:|\n",
    "|PERSON\t| People, including fictional. |\n",
    "|NORP\t| Nationalities or religious or political groups. |\n",
    "|FAC\t|Buildings, airports, highways, bridges, etc.|\n",
    "|ORG\t|Companies, agencies, institutions, etc.|\n",
    "|GPE\t|Countries, cities, states.|\n",
    "|LOC\t|Non-GPE locations, mountain ranges, bodies of water.|\n",
    "|PRODUCT|\tObjects, vehicles, foods, etc. (Not services.)|\n",
    "|EVENT\t|Named hurricanes, battles, wars, sports events, etc.|\n",
    "|WORK_OF_ART|\tTitles of books, songs, etc.|\n",
    "|LAW\t|Named documents made into laws.|\n",
    "|LANGUAGE|\tAny named language.|\n",
    "|DATE\t|Absolute or relative dates or periods.|\n",
    "|TIME\t|Times smaller than a day.|\n",
    "|ERCENT\t|Percentage, including '%'.|\n",
    "|MONEY\t|Monetary values, including unit.|\n",
    "|QUANTITY|\tMeasurements, as of weight or distance.|\n",
    "|ORDINAL|\t“first”, “second”, etc.|\n",
    "|CARDINAL|\tNumerals that do not fall under another type.|\n",
    "\n",
    "We would like to add our own new entity tag to this list for items related to 'machine learning', which we will give the tag of 'ML'. <code>spaCy</code> provides two methods for extending entity recognition: 1) training and updating an existing model (in this case, <code>en_core_web_sm</code>); or 2) a rule-based approach.\n",
    "\n",
    "The first option we will try is the updated model. Another major section below investigates the rule-based approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Get Entity Pages\n",
    "We already have a number of 'machine learning'-related reference concepts in KBpedia. Given our success in using the Wikipedia online API to get articles (see prior installment), I decide to explore that API to see if there are ways to get comprehensive listings of 'machine learning' topics. Happily, it turns out, there are!\n",
    "\n",
    "This particular API call, https://www.mediawiki.org/wiki/API:Categorymembers, enables us to enter a category term, in this case 'machine learning', and to get all of the article titles subsumed in that category. What is also fantastic is that we can also get various code snippets to interact with this API, including using Python. Here is the code listing that we get from the API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/python3\n",
    "\n",
    "\"\"\"\n",
    "    get_category_items.py\n",
    "\n",
    "    MediaWiki API Demos\n",
    "    Derived from demo of `Categorymembers` module : List twenty items in a category\n",
    "\n",
    "    MIT License\n",
    "\"\"\"\n",
    "\n",
    "import requests\n",
    "\n",
    "S = requests.Session()\n",
    "\n",
    "URL = \"https://en.wikipedia.org/w/api.php\"\n",
    "\n",
    "PARAMS = {\n",
    "    \"action\": \"query\",\n",
    "    \"cmtitle\": \"Category:Unsupervised learning\",\n",
    "    \"cmlimit\": \"300\",\n",
    "    \"list\": \"categorymembers\",\n",
    "    \"format\": \"json\"\n",
    "}\n",
    "\n",
    "R = S.get(url=URL, params=PARAMS)\n",
    "DATA = R.json()\n",
    "\n",
    "PAGES = DATA['query']['categorymembers']\n",
    "\n",
    "for page in PAGES:\n",
    "    print(page['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each appropriate sub-category under 'machine learning' we issue the query above to get a listing of possible articles on the topic in Wikipedia. We assemble up this listing and then manually inspect it to remove things like <code>Category:Machine learning researchers</code>, since those are people related to machine learning but not 'machine learning' *per se*. The result of this retrieval, including all relevant subdirectories, yields 1027 pages, 853 of which are unique, and 846 of which actually process.\n",
    "\n",
    "This listing gives us two kinds of items. First, the page titles give us terms and phrases related to 'machine learning'. Second, using the same procedures for Wikipedia page retrievals noted in [**CWPK #63**](https://www.mkbergman.com/2414/cwpk-63-staging-data-sci-resources-and-preprocessing/), we retrieve the actual XML pages, clean them in the same way we did for the general KBpedia corpus, and then create bigrams and trigrams. We now have our specialty 'machine learning' corpus in the exact same format as that for KBpedia, which we keep and save as <code>wp_machine_learning.txt</code>.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Chunk Text into 'Sentences'\n",
    "Since the <code>spaCy</code> NER trainer relies on sentence-length snippets for its training (see below), our next step is to chop up this text corpus into sentence-length chunks. The code below, including the <code>textwrap</code> package import, enables us to iterate document-by-document through our <code>wp_machine_learning.txt</code> file and to break it into sentence-length chunks. The example below chunks into snippets that are 48 characters long:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textwrap\n",
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wp_machine_learning.txt'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wp_machine_learning_sentences_48.txt'\n",
    "\n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "    i = 0\n",
    "    for line in documents:\n",
    "        try:\n",
    "            line = str(line)\n",
    "            sentences = textwrap.wrap(line, 48)\n",
    "            sentences = str(sentences)\n",
    "            sentences = sentences + '\\n'\n",
    "            output.write(sentences)\n",
    "        except Exception as e:\n",
    "            print ('Exception error: ' + str(e))\n",
    "        i = i + 1\n",
    "    output.close()\n",
    "    print('Split lines into sentences for ' + str(i) + ' articles;')\n",
    "    print('Processing complete!')   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my various experiments, I set chunk sizes ranging from 48 characters to 180 characters, as I discuss in later results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extract and Offset Entities\n",
    "The titles we extracted from Wikipedia 'machine learning' articles give us the set terms and phrases for setting up the labeled examples expected by the <code>spaCy</code> NER trainer. Here is one example:\n",
    "\n",
    "   <pre>\"a generative model it is now known as a\", {\"entities\": [(2, 18, \"ML\")]}</pre>\n",
    "\n",
    "This labeled example shows a text pattern in which the entity ('generative model' in this case) is embedded, with the starting and ending character offsets specified for that entity, as well as its 'ML' label. (Note that the offset counter begins at zero given the Python convention.) One needs to provide hundreds of such labeled examples to properly train the entity recognizer.\n",
    "\n",
    "Manually labeling these snippets is an intense and time-consuming task. To make it efficient, we use a <code>spaCy</code> function called <code>PhraseMatcher</code> that inspects each snippet, identifies whether a stipulated entity occurs there, and returns the offset character number where it matches. Thus, in the code below, we first list out the 800 or so 'machine learning' titles we have already identified, and then parse those against the sentence snippets we generated from our article texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adapted from https://stackoverflow.com/questions/47638877/using-phrasematcher-in-spacy-to-find-multiple-match-types\n",
    "# https://thinkinfi.com/prepare-training-data-and-train-custom-ner-using-spacy-python/\n",
    "# https://adagrad.ai/blog/ner-using-spacy.html\n",
    "\n",
    "import os\n",
    "import spacy\n",
    "import random\n",
    "from spacy.matcher import PhraseMatcher\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from smart_open import smart_open\n",
    "import en_core_web_sm\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wp_machine_learning_sentences_48.txt'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wp_machine_learning_training_data_48.json'\n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "ml_patterns = [nlp(text) for text in ('(1+ε)-approximate nearest neighbor search', \n",
    "     '80 million tiny images', 'ablation', 'absorbing markov chain')]\n",
    "# See the full listing under '(Named) Entity Recognition - Rule-Based' section below                                     \n",
    "\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "matcher.add('ML', None, *ml_patterns)\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "    x = 0\n",
    "    for line in documents:\n",
    "        line = str(line)\n",
    "        sublist = line.split(', ')\n",
    "        for sentence in sublist:\n",
    "            sentence = str(sentence)\n",
    "            sentence = sentence.replace(']','')\n",
    "            sentence = sentence.replace(\"'\", \"\")\n",
    "            doc = nlp(sentence)\n",
    "            sen_length = len(sentence)\n",
    "            matches = matcher(doc)\n",
    "            start = 0\n",
    "            s_start = 0\n",
    "            s_length = 0\n",
    "            for match_id, start, end in matches:  # iterate over the entities\n",
    "                label = nlp.vocab.strings[match_id]\n",
    "                span = doc[start:end]\n",
    "                label = str(label)\n",
    "                span = str(span)\n",
    "                length = len(span)\n",
    "                start = sentence.index(span)\n",
    "                end = start + length\n",
    "                start = str(start)\n",
    "                end = str(end)\n",
    "                if s_start != start and s_length != length:\n",
    "                    s_start = start\n",
    "                    s_length = length\n",
    "                    train_it = ('(\"' + sentence + '\", {\"entities\": [(' + start + ', ' + end \n",
    "                           + ', \"' + label + '\")]}),')\n",
    "                    output.write(train_it)\n",
    "                else:\n",
    "                    continue\n",
    "    output.close()\n",
    "    print('Got this far!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code block at the end of this routine calculates the starting and ending character offsets for the matched entity, and then constructs up a new string that matches the form expected by <code>spaCy</code>.  Note this training data needs to be in JSON format. There are online JSON syntax checkers (here is [one](https://www.freeformatter.com/json-validator.html)) to make sure you are constructing this training example in the correct form. It takes about 8 min to do the conversion above when parsed against our machine learning corpus.\n",
    "\n",
    "##### Train the Recognizer\n",
    "Re-training an existing <code>spaCy</code> NER model means to import the existing model and update the model using the training example snippets. However, if not done properly, one can experience what is called the '[catastrophic forgetting problem](https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting)', which means that existing trained labels get forgotten as the new ones are learned. Two steps are recommended to limit this problem. First, the training should be limited to about 20 iterations, since repeated iterations risk more forgetting. The second recommended step is to include existing label snippets in the training corpus when training begins. This way the existing label is seen again, and the degree of 'forgetting' is lessened.\n",
    "\n",
    "I looked in vain for finding the existing training examples used by <code>spaCy</code> for its existing entity labels. Not having success in finding such, I decided to create my own snippets with existing labels.\n",
    "\n",
    "To do so, I repeated similar steps to what was outlined above, only now to use the existing labels rather than new ones. Like before, we also need to construct the training example in the proper JSON form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import spacy\n",
    "import random\n",
    "from spacy.gold import GoldParse\n",
    "\n",
    "from smart_open import smart_open\n",
    "import en_core_web_sm\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wp_machine_learning_sentences_48.txt'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\standard_training_data_48.json'\n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "nlp = en_core_web_sm.load()\n",
    "revision_data = []\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "#    for doc in nlp.pipe(revision_texts):\n",
    "    for line in documents:\n",
    "        line = str(line)\n",
    "        sublist = line.split(', ')\n",
    "        for sentence in sublist:\n",
    "            sentence = str(sentence)\n",
    "            sentence = sentence.replace(']','')\n",
    "            sentence = sentence.replace(\"'\", \"\")\n",
    "            sentence = sentence.replace('[','')\n",
    "            sentence = sentence.replace('\\n', '')\n",
    "            length = len(sentence)\n",
    "            if length < 40:\n",
    "                continue\n",
    "            else:\n",
    "                doc = nlp(sentence)\n",
    "#        tags = [w.tag_ for w in doc]\n",
    "#        heads = [w.head.i for w in doc]\n",
    "#        deps = [w.dep_ for w in doc]\n",
    "                entities = [(e.start_char, e.end_char, e.label_) for e in doc.ents]\n",
    "#        revision_data.append((doc, GoldParse(doc, tags=tags, heads=heads,\n",
    "#                                            deps=deps, entities=entities)))\n",
    "                revision_data.append((doc, GoldParse(doc, entities=entities)))\n",
    "#        print(revision_data)\n",
    "                doc = str(doc)\n",
    "                entities = str(entities)\n",
    "#            revision_str = (doc + ', ' + entities + '\\n')\n",
    "                revision_str = ('(\"' + doc + '\", {\"entities\": ' + entities + '}),\\n')\n",
    "                output.write(revision_str)\n",
    "    output.close()\n",
    "    print('Got this far!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This second pass using the existing NER labels took about 19 min.\n",
    "\n",
    "The last step in our prep for updating the existing NER model is to remove short stubs from our training examples, as this code achieves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\standard_training_data_48.json'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\standard_training_data.json'\n",
    "\n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "    i = 0\n",
    "    for line in documents:\n",
    "        try:\n",
    "            line = str(line)\n",
    "            length = len(line)\n",
    "            if i == length or length < 40:\n",
    "                continue\n",
    "            else:\n",
    "                i = length\n",
    "                output.write(line)\n",
    "        except Exception as e:\n",
    "            print ('Exception error: ' + str(e))\n",
    "    output.close()\n",
    "    print('Processing complete!')   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Continue Training the Recognizer\n",
    "\n",
    "We add about 10% of existing label training examples to those we have already generated for the 'ML' training set to overcome the 'catastrophic forgetting failure'. With this new input deck, we are now ready to run and update the NER recognizer, making sure to keep our iterations below 20 epochs. Here is the code, including some of the 14 K training examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adapted from https://www.machinelearningplus.com/nlp/training-custom-ner-model-in-spacy/\n",
    "\n",
    "# Import and load the spacy model\n",
    "import spacy\n",
    "import en_core_web_sm\n",
    "import random\n",
    "from spacy.util import minibatch, compounding\n",
    "\n",
    "output_dir = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\en_core_ml'\n",
    " \n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "# Get the ner component\n",
    "ner=nlp.get_pipe('ner')\n",
    "\n",
    "# Add new label\n",
    "LABEL = 'ML'\n",
    "\n",
    "# Add the label to ner\n",
    "ner.add_label(LABEL)\n",
    "\n",
    "# Load training data \n",
    "TRAIN_DATA = [(\"ökonomik path dependence in spatial networks the\", {\"entities\": [(9, 24, \"ML\")]}),\n",
    "(\"β with respect to the loss function v if the\", {\"entities\": [(22, 35, \"ML\")]}),\n",
    "(\"ε approximate nearest neighbor search include kd\", {\"entities\": [(14, 37, \"ML\")]}),\n",
    "(\"ξ i e φ x is the feature vector produced for a\", {\"entities\": [(17, 24, \"ML\")]}),\n",
    "(\"a a survey on concept drift adaptation acm\", {\"entities\": [(14, 27, \"ML\")]}),\n",
    "(\"a bayesian gaussian mixture model is commonly\", {\"entities\": [(20, 33, \"ML\")]}),\n",
    "(\"a beginner s guide to factor analysis\", {\"entities\": [(22, 37, \"ML\")]}),\n",
    "(\"a boltzmann machine with a few weights labeled\", {\"entities\": [(2, 19, \"ML\")]}),\n",
    "# See 'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\ml_training_data.json' for full listing\n",
    "\n",
    "# Retrieve labels\n",
    "for _, annotations in TRAIN_DATA:\n",
    "    for ent in annotations.get('entities'):\n",
    "        ner.add_label(ent[2])\n",
    "            \n",
    "# Resume training (since we are extending an existing model)\n",
    "optimizer = nlp.resume_training()\n",
    "move_names = list(ner.move_names)\n",
    "\n",
    "# List pipes needed for training\n",
    "pipe_exceptions = ['ner', 'trf_wordpiecer', 'trf_tok2vec']\n",
    "\n",
    "# List all other pipes\n",
    "other_pipes = [pipe for pipe in nlp.pipe_names if pipe not in pipe_exceptions]  \n",
    "\n",
    "# Begin training on restricted pipeline components\n",
    "with nlp.disable_pipes(*other_pipes):\n",
    "    random.seed(0)\n",
    "    sizes = compounding(1.0, 4.0, 1.001)\n",
    "# Iterate over training set 20 time     \n",
    "    for itn in range(20):\n",
    "# Shuffle examples\n",
    "        random.shuffle(TRAIN_DATA)\n",
    "# Grab a batch of examples for training\n",
    "        batches = minibatch(TRAIN_DATA, size=sizes)\n",
    "# Set empty dictionary\n",
    "        losses = {}\n",
    "        for batch in batches:\n",
    "            texts, annotations = zip(*batch)\n",
    "# Call update() over epoch; see narrative\n",
    "            nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)\n",
    "            print('Losses: ', losses)\n",
    "# Save model to disk\n",
    "    nlp.meta['name'] = 'ml_sm'  # rename model\n",
    "    nlp.to_disk(output_dir)\n",
    "    print('Saved model to: ', output_dir) \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When run, a loss calculation is made for each training input example, which gets repeated in full over the number of iterations specified (20 in this case). (Which takes about 4:15 hrs to run on my laptop.) Also note we are saving out the new trained model, <code>en_core_ml</code>, at the completion of the training.\n",
    "\n",
    "Unfortunately, though my testing showed we were picking up the 'ML' tag, it was not doing so for many of the input training examples. To see if I could improve on this performance, I tested the following options, all to no material benefit:\n",
    "\n",
    "- Reducing the number of training examples\n",
    "- Reducing the number of entity patterns\n",
    "- Varying the sentence snippet size, from 48 characters (more typical of online examples) to 180 characters (more typical of actual sentence length)\n",
    "- Dialing back the number of iterations (even a single iteration showed the forgetting behavior)\n",
    "- Testing the <code>drop</code> setting between 0.5 and 0.0\n",
    "- Changing the relative contribution of existing NER labels to the training set\n",
    "- Using or not negative (no entity matches) training examples.\n",
    "\n",
    "In all cases, I continued to see the 'forgetting' problem and observed that larger numbers of iterations adversely affected the number of 'ML' labels assigned. These results are not in accordance with the documentation I have found. Possible reasons for this continued poor performance might include:\n",
    "\n",
    "- Too many diverse patterns in the training set\n",
    "- Issues possibly arising from the synthetic sentence snippets I generated\n",
    "- An inadquate percentage of existing label training snippets, or\n",
    "- A coding error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Named) Entity Recognition - Rule-Based\n",
    "Unlike certain named entities like persons or organizations or locations, the number of 'machine learning' instances is more bounded and finite. Since I had already captured the nearly complete entity aspects of the space through the thousand Wikipedia examples, perhaps I did not need a trained model, but one more based on rules and set patterns. \n",
    "\n",
    "Fortunately, <code>spaCy</code> has such a capability called [EntityRuler](https://spacy.io/api/entityruler). It provides [rule-based matching](https://spacy.io/usage/rule-based-matching#entityruler) for assigning label tags to text. Given that I had been unable to better tune the training model, I decided to test this option as well.\n",
    "\n",
    "Like the training models, the rule-based approach records a set of known input patterns to provide the matches to text and then labeling. So, aside from the need to assemble the known patterns, which we had already done above, the actual code to invoke this option is rather simple and straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved model to:  C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\en_core_ml\n"
     ]
    }
   ],
   "source": [
    "from spacy.pipeline import EntityRuler\n",
    "import en_core_web_sm\n",
    "\n",
    "output_dir = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\en_core_ml'\n",
    " \n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "ruler = EntityRuler(nlp, overwrite_ents=True)\n",
    "\n",
    "patterns = [{'label': 'ML', 'pattern': 'nearest neighbor search'},\n",
    "{'label': 'ML', 'pattern': '80 million tiny images'},\n",
    "{'label': 'ML', 'pattern': 'ablation'},\n",
    "{'label': 'ML', 'pattern': 'absorbing markov chain'},\n",
    "{'label': 'ML', 'pattern': 'action model learning'},\n",
    "{'label': 'ML', 'pattern': 'activation function'},\n",
    "{'label': 'ML', 'pattern': 'active learning'},\n",
    "{'label': 'ML', 'pattern': 'activity recognition'},\n",
    "{'label': 'ML', 'pattern': 'adaboost'},\n",
    "{'label': 'ML', 'pattern': 'adagrad'},\n",
    "{'label': 'ML', 'pattern': 'adaline'},\n",
    "{'label': 'ML', 'pattern': 'adaptive neuro fuzzy inference system'},\n",
    "{'label': 'ML', 'pattern': 'adaptive resonance theory'},\n",
    "{'label': 'ML', 'pattern': 'additive smoothing'},\n",
    "{'label': 'ML', 'pattern': 'adversarial machine learning'},\n",
    "{'label': 'ML', 'pattern': 'aixi'},\n",
    "{'label': 'ML', 'pattern': 'alchemyapi'},\n",
    "{'label': 'ML', 'pattern': 'alexnet'},\n",
    "{'label': 'ML', 'pattern': 'algorithm selection'},\n",
    "{'label': 'ML', 'pattern': 'algorithmic bias'},\n",
    "{'label': 'ML', 'pattern': 'algorithmic composition'},\n",
    "{'label': 'ML', 'pattern': 'algorithmic inference'},\n",
    "{'label': 'ML', 'pattern': 'algorithmic learning theory'},\n",
    "{'label': 'ML', 'pattern': 'algorithms of oppression'},\n",
    "{'label': 'ML', 'pattern': 'almeida–pineda recurrent backpropagation'},\n",
    "{'label': 'ML', 'pattern': 'alopex'},\n",
    "{'label': 'ML', 'pattern': 'alphago'},\n",
    "{'label': 'ML', 'pattern': 'alphago zero'},\n",
    "{'label': 'ML', 'pattern': 'alphastar'},\n",
    "{'label': 'ML', 'pattern': 'alphazero'},\n",
    "{'label': 'ML', 'pattern': 'alterego'},\n",
    "{'label': 'ML', 'pattern': 'alternating decision tree'},\n",
    "{'label': 'ML', 'pattern': 'analogical modeling'},\n",
    "{'label': 'ML', 'pattern': 'anomaly detection'},\n",
    "{'label': 'ML', 'pattern': 'anti-unification'},\n",
    "{'label': 'ML', 'pattern': 'apprenticeship learning'},\n",
    "{'label': 'ML', 'pattern': 'archetypal analysis'},\n",
    "{'label': 'ML', 'pattern': 'artificial development'},\n",
    "{'label': 'ML', 'pattern': 'artificial intelligence system'},\n",
    "{'label': 'ML', 'pattern': 'artificial neural network'},\n",
    "{'label': 'ML', 'pattern': 'artificial neuron'},\n",
    "{'label': 'ML', 'pattern': 'artisto'},\n",
    "{'label': 'ML', 'pattern': 'associative classifier'},\n",
    "{'label': 'ML', 'pattern': 'astrostatistics'},\n",
    "{'label': 'ML', 'pattern': 'augmented analytics'},\n",
    "{'label': 'ML', 'pattern': 'autoassociative memory'},\n",
    "{'label': 'ML', 'pattern': 'autoencoder'},\n",
    "{'label': 'ML', 'pattern': 'automated machine learning'},\n",
    "{'label': 'ML', 'pattern': 'automated pain recognition'},\n",
    "{'label': 'ML', 'pattern': 'averaged one-dependence estimators'},\n",
    "{'label': 'ML', 'pattern': 'backpropagation'},\n",
    "{'label': 'ML', 'pattern': 'bag-of-words'},\n",
    "{'label': 'ML', 'pattern': 'ball tree'},\n",
    "{'label': 'ML', 'pattern': 'base rate'},\n",
    "{'label': 'ML', 'pattern': 'baum–welch algorithm'},\n",
    "{'label': 'ML', 'pattern': 'bayesian hierarchical modeling'},\n",
    "{'label': 'ML', 'pattern': 'bayesian interpretation of kernel regularization'},\n",
    "{'label': 'ML', 'pattern': 'bayesian network'},\n",
    "{'label': 'ML', 'pattern': 'bayesian optimization'},\n",
    "{'label': 'ML', 'pattern': 'bayesian regret'},\n",
    "{'label': 'ML', 'pattern': 'bayesian structural time series'},\n",
    "{'label': 'ML', 'pattern': 'bcpnn'},\n",
    "{'label': 'ML', 'pattern': 'behavioral clustering'},\n",
    "{'label': 'ML', 'pattern': 'bernoulli scheme'},\n",
    "{'label': 'ML', 'pattern': 'bias–variance tradeoff'},\n",
    "{'label': 'ML', 'pattern': 'biclustering'},\n",
    "{'label': 'ML', 'pattern': 'bidirectional associative memory'},\n",
    "{'label': 'ML', 'pattern': 'bidirectional recurrent neural networks'},\n",
    "{'label': 'ML', 'pattern': 'binary classification'},\n",
    "{'label': 'ML', 'pattern': 'bioz'},\n",
    "{'label': 'ML', 'pattern': 'boltzmann machine'},\n",
    "{'label': 'ML', 'pattern': 'bondys theorem'},\n",
    "{'label': 'ML', 'pattern': 'bongard problem'},\n",
    "{'label': 'ML', 'pattern': 'boosting'},\n",
    "{'label': 'ML', 'pattern': 'bootstrap aggregating'},\n",
    "{'label': 'ML', 'pattern': 'bradley–terry model'},\n",
    "{'label': 'ML', 'pattern': 'brown clustering'},\n",
    "{'label': 'ML', 'pattern': 'brownboost'},\n",
    "{'label': 'ML', 'pattern': 'burst error'},\n",
    "{'label': 'ML', 'pattern': 'c4.5 algorithm'},\n",
    "{'label': 'ML', 'pattern': 'calibration'},\n",
    "{'label': 'ML', 'pattern': 'canonical correspondence analysis'},\n",
    "{'label': 'ML', 'pattern': 'capsule neural network'},\n",
    "{'label': 'ML', 'pattern': 'cartesian genetic programming'},\n",
    "{'label': 'ML', 'pattern': 'cascading classifiers'},\n",
    "{'label': 'ML', 'pattern': 'case-based reasoning'},\n",
    "{'label': 'ML', 'pattern': 'catastrophic interference'},\n",
    "{'label': 'ML', 'pattern': 'category utility'},\n",
    "{'label': 'ML', 'pattern': 'causal markov condition'},\n",
    "{'label': 'ML', 'pattern': 'cellular evolutionary algorithm'},\n",
    "{'label': 'ML', 'pattern': 'cellular neural network'},\n",
    "{'label': 'ML', 'pattern': 'cerebellar model articulation controller'},\n",
    "{'label': 'ML', 'pattern': 'chainer'},\n",
    "{'label': 'ML', 'pattern': 'chi-square automatic interaction detection'},\n",
    "{'label': 'ML', 'pattern': 'classifier chains'},\n",
    "{'label': 'ML', 'pattern': 'clever score'},\n",
    "{'label': 'ML', 'pattern': 'cluster analysis'},\n",
    "{'label': 'ML', 'pattern': 'clustering high-dimensional data'},\n",
    "{'label': 'ML', 'pattern': 'clustering illusion'},\n",
    "{'label': 'ML', 'pattern': 'cma-es'},\n",
    "{'label': 'ML', 'pattern': 'cn2 algorithm'},\n",
    "{'label': 'ML', 'pattern': 'co-training'},\n",
    "{'label': 'ML', 'pattern': 'coboosting'},\n",
    "{'label': 'ML', 'pattern': 'codi'},\n",
    "{'label': 'ML', 'pattern': 'cognitive computer'},\n",
    "{'label': 'ML', 'pattern': 'cognitive robotics'},\n",
    "{'label': 'ML', 'pattern': 'collostructional analysis'},\n",
    "{'label': 'ML', 'pattern': 'committee machine'},\n",
    "{'label': 'ML', 'pattern': 'common-method variance'},\n",
    "{'label': 'ML', 'pattern': 'competitive learning'},\n",
    "{'label': 'ML', 'pattern': 'compositional pattern-producing network'},\n",
    "{'label': 'ML', 'pattern': 'computational cybernetics'},\n",
    "{'label': 'ML', 'pattern': 'computational learning theory'},\n",
    "{'label': 'ML', 'pattern': 'computational neurogenetic modeling'},\n",
    "{'label': 'ML', 'pattern': 'computer-automated design'},\n",
    "{'label': 'ML', 'pattern': 'concept class'},\n",
    "{'label': 'ML', 'pattern': 'concept drift'},\n",
    "{'label': 'ML', 'pattern': 'concept learning'},\n",
    "{'label': 'ML', 'pattern': 'conceptual clustering'},\n",
    "{'label': 'ML', 'pattern': 'conditional random field'},\n",
    "{'label': 'ML', 'pattern': 'confabulation'},\n",
    "{'label': 'ML', 'pattern': 'confusion matrix'},\n",
    "{'label': 'ML', 'pattern': 'connectionist temporal classification'},\n",
    "{'label': 'ML', 'pattern': 'consensus clustering'},\n",
    "{'label': 'ML', 'pattern': 'constellation model'},\n",
    "{'label': 'ML', 'pattern': 'constrained clustering'},\n",
    "{'label': 'ML', 'pattern': 'constrained conditional model'},\n",
    "{'label': 'ML', 'pattern': 'constructing skill trees'},\n",
    "{'label': 'ML', 'pattern': 'constructive cooperative coevolution'},\n",
    "{'label': 'ML', 'pattern': 'conversica'},\n",
    "{'label': 'ML', 'pattern': 'convolutional deep belief network'},\n",
    "{'label': 'ML', 'pattern': 'convolutional neural network', 'id': 'cnn'},\n",
    "{'label': 'ML', 'pattern': 'CNN', 'id': 'cnn'},            \n",
    "{'label': 'ML', 'pattern': 'correlation clustering'},\n",
    "{'label': 'ML', 'pattern': 'correspondence analysis'},\n",
    "{'label': 'ML', 'pattern': 'count sketch'},\n",
    "{'label': 'ML', 'pattern': 'coupled pattern learner'},\n",
    "{'label': 'ML', 'pattern': 'covers theorem'},\n",
    "{'label': 'ML', 'pattern': 'cross entropy'},\n",
    "{'label': 'ML', 'pattern': 'cross-validation'},\n",
    "{'label': 'ML', 'pattern': 'cultural algorihm'},\n",
    "{'label': 'ML', 'pattern': 'curse of dimensionality'},\n",
    "{'label': 'ML', 'pattern': 'darkforest'},\n",
    "{'label': 'ML', 'pattern': 'darpa lagr program'},\n",
    "{'label': 'ML', 'pattern': 'darwintunes'},\n",
    "{'label': 'ML', 'pattern': 'data augmentation'},\n",
    "{'label': 'ML', 'pattern': 'data exploration'},\n",
    "{'label': 'ML', 'pattern': 'data pre-processing'},\n",
    "{'label': 'ML', 'pattern': 'datasets.load'},\n",
    "{'label': 'ML', 'pattern': 'decision boundary'},\n",
    "{'label': 'ML', 'pattern': 'decision list'},\n",
    "{'label': 'ML', 'pattern': 'decision tree learning'},\n",
    "{'label': 'ML', 'pattern': 'decision tree pruning'},\n",
    "{'label': 'ML', 'pattern': 'deductive classifier'},\n",
    "{'label': 'ML', 'pattern': 'deep belief network'},\n",
    "{'label': 'ML', 'pattern': 'deep image prior'},\n",
    "{'label': 'ML', 'pattern': 'deep instinct'},\n",
    "{'label': 'ML', 'pattern': 'deep lambertian networks'},\n",
    "{'label': 'ML', 'pattern': 'deep learning'},\n",
    "{'label': 'ML', 'pattern': 'deep learning processor'},\n",
    "{'label': 'ML', 'pattern': 'deep learning studio'},\n",
    "{'label': 'ML', 'pattern': 'deep reinforcement learning'},\n",
    "{'label': 'ML', 'pattern': 'deepfake'},\n",
    "{'label': 'ML', 'pattern': 'deepfake pornography'},\n",
    "{'label': 'ML', 'pattern': 'deeplearning4j'},\n",
    "{'label': 'ML', 'pattern': 'deepmind'},\n",
    "{'label': 'ML', 'pattern': 'deepnude'},\n",
    "{'label': 'ML', 'pattern': 'deepspeed'},\n",
    "{'label': 'ML', 'pattern': 'dehaene–changeux model'},\n",
    "{'label': 'ML', 'pattern': 'delta rule'},\n",
    "{'label': 'ML', 'pattern': 'dendrogram'},\n",
    "{'label': 'ML', 'pattern': 'dependability state model'},\n",
    "{'label': 'ML', 'pattern': 'detailed balance'},\n",
    "{'label': 'ML', 'pattern': 'detrended correspondence analysis'},\n",
    "{'label': 'ML', 'pattern': 'developmental robotics'},\n",
    "{'label': 'ML', 'pattern': 'dexnet'},\n",
    "{'label': 'ML', 'pattern': 'diffbot'},\n",
    "{'label': 'ML', 'pattern': 'differentiable neural computer'},\n",
    "{'label': 'ML', 'pattern': 'differential evolution'},\n",
    "{'label': 'ML', 'pattern': 'diffusion map'},\n",
    "{'label': 'ML', 'pattern': 'dimensionality reduction'},\n",
    "{'label': 'ML', 'pattern': 'discrete phase-type distribution'},\n",
    "{'label': 'ML', 'pattern': 'discriminative model'},\n",
    "{'label': 'ML', 'pattern': 'dispersive flies optimisation'},\n",
    "{'label': 'ML', 'pattern': 'dissociated press'},\n",
    "{'label': 'ML', 'pattern': 'distribution learning theory'},\n",
    "{'label': 'ML', 'pattern': 'document classification'},\n",
    "{'label': 'ML', 'pattern': 'domain adaptation'},\n",
    "{'label': 'ML', 'pattern': 'dominance-based rough set approach'},\n",
    "{'label': 'ML', 'pattern': 'doubly stochastic model'},\n",
    "{'label': 'ML', 'pattern': 'dynamic bayesian network'},\n",
    "{'label': 'ML', 'pattern': 'dynamic markov compression'},\n",
    "{'label': 'ML', 'pattern': 'dynamic time warping'},\n",
    "{'label': 'ML', 'pattern': 'dynamic topic model'},\n",
    "{'label': 'ML', 'pattern': 'dynamic unobserved effects model'},\n",
    "{'label': 'ML', 'pattern': 'eager learning'},\n",
    "{'label': 'ML', 'pattern': 'early stopping'},\n",
    "{'label': 'ML', 'pattern': 'echo state network'},\n",
    "{'label': 'ML', 'pattern': 'effective fitness'},\n",
    "{'label': 'ML', 'pattern': 'elastic map'},\n",
    "{'label': 'ML', 'pattern': 'elastic matching'},\n",
    "{'label': 'ML', 'pattern': 'elastic net regularization'},\n",
    "{'label': 'ML', 'pattern': 'electricity price forecasting'},\n",
    "{'label': 'ML', 'pattern': 'elmo'},\n",
    "{'label': 'ML', 'pattern': 'em algorithm and gmm model'},\n",
    "{'label': 'ML', 'pattern': 'empirical risk minimization'},\n",
    "{'label': 'ML', 'pattern': 'end-to-end reinforcement learning'},\n",
    "{'label': 'ML', 'pattern': 'ensemble learning'},\n",
    "{'label': 'ML', 'pattern': 'entropy rate'},\n",
    "{'label': 'ML', 'pattern': 'error tolerance'},\n",
    "{'label': 'ML', 'pattern': 'error-driven learning'},\n",
    "{'label': 'ML', 'pattern': 'eurisko'},\n",
    "{'label': 'ML', 'pattern': 'european neural network society'},\n",
    "{'label': 'ML', 'pattern': 'evaluation of binary classifiers'},\n",
    "{'label': 'ML', 'pattern': 'evolution strategy'},\n",
    "{'label': 'ML', 'pattern': 'evolution window'},\n",
    "{'label': 'ML', 'pattern': 'evolutionary algorithm'},\n",
    "{'label': 'ML', 'pattern': 'evolutionary art'},\n",
    "{'label': 'ML', 'pattern': 'evolutionary multimodal optimization'},\n",
    "{'label': 'ML', 'pattern': 'evolutionary music'},\n",
    "{'label': 'ML', 'pattern': 'evolutionary programming'},\n",
    "{'label': 'ML', 'pattern': 'evolvability'},\n",
    "{'label': 'ML', 'pattern': 'evolved antenna'},\n",
    "{'label': 'ML', 'pattern': 'evolving classification function'},\n",
    "{'label': 'ML', 'pattern': 'examples of markov chains'},\n",
    "{'label': 'ML', 'pattern': 'expectation propagation'},\n",
    "{'label': 'ML', 'pattern': 'expectation–maximization algorithm'},\n",
    "{'label': 'ML', 'pattern': 'explanation-based learning'},\n",
    "{'label': 'ML', 'pattern': 'extension neural network'},\n",
    "{'label': 'ML', 'pattern': 'extremal ensemble learning'},\n",
    "{'label': 'ML', 'pattern': 'extreme learning machine'},\n",
    "{'label': 'ML', 'pattern': 'f-score'},\n",
    "{'label': 'ML', 'pattern': 'faceapp'},\n",
    "{'label': 'ML', 'pattern': 'facial recognition system'},\n",
    "{'label': 'ML', 'pattern': 'factor analysis'},\n",
    "{'label': 'ML', 'pattern': 'factor regression model'},\n",
    "{'label': 'ML', 'pattern': 'factored language model'},\n",
    "{'label': 'ML', 'pattern': 'farthest-first traversal'},\n",
    "{'label': 'ML', 'pattern': 'feature engineering'},\n",
    "{'label': 'ML', 'pattern': 'feature extraction'},\n",
    "{'label': 'ML', 'pattern': 'feature hashing'},\n",
    "{'label': 'ML', 'pattern': 'feature learning'},\n",
    "{'label': 'ML', 'pattern': 'feature scaling'},\n",
    "{'label': 'ML', 'pattern': 'feature selection'},\n",
    "{'label': 'ML', 'pattern': 'feature selection toolbox'},\n",
    "{'label': 'ML', 'pattern': 'federated learning'},\n",
    "{'label': 'ML', 'pattern': 'feed forward'},\n",
    "{'label': 'ML', 'pattern': 'feedforward neural network'},\n",
    "{'label': 'ML', 'pattern': 'feret database'},\n",
    "{'label': 'ML', 'pattern': 'findface'},\n",
    "{'label': 'ML', 'pattern': 'first-difference estimator'},\n",
    "{'label': 'ML', 'pattern': 'first-order inductive learner'},\n",
    "{'label': 'ML', 'pattern': 'fisher kernel'},\n",
    "{'label': 'ML', 'pattern': 'fitness approximation'},\n",
    "{'label': 'ML', 'pattern': 'fly algorithm'},\n",
    "{'label': 'ML', 'pattern': 'formal concept analysis'},\n",
    "{'label': 'ML', 'pattern': 'forward algorithm'},\n",
    "{'label': 'ML', 'pattern': 'forward–backward algorithm'},\n",
    "{'label': 'ML', 'pattern': 'frequent pattern discovery'},\n",
    "{'label': 'ML', 'pattern': 'gated recurrent unit'},\n",
    "{'label': 'ML', 'pattern': 'gaussian adaptation'},\n",
    "{'label': 'ML', 'pattern': 'gaussian process'},\n",
    "{'label': 'ML', 'pattern': 'gaussian process emulator'},\n",
    "{'label': 'ML', 'pattern': 'gene expression programming'},\n",
    "{'label': 'ML', 'pattern': 'gene prediction'},\n",
    "{'label': 'ML', 'pattern': 'general regression neural network'},\n",
    "{'label': 'ML', 'pattern': 'generalization error'},\n",
    "{'label': 'ML', 'pattern': 'generalized canonical correlation'},\n",
    "{'label': 'ML', 'pattern': 'generalized filtering'},\n",
    "{'label': 'ML', 'pattern': 'generalized hebbian algorithm'},\n",
    "{'label': 'ML', 'pattern': 'generalized iterative scaling'},\n",
    "{'label': 'ML', 'pattern': 'generalized multidimensional scaling'},\n",
    "{'label': 'ML', 'pattern': 'generative adversarial network'},\n",
    "{'label': 'ML', 'pattern': 'generative model'},\n",
    "{'label': 'ML', 'pattern': 'generative topographic map'},\n",
    "{'label': 'ML', 'pattern': 'generec'},\n",
    "{'label': 'ML', 'pattern': 'genetic algorithm'},\n",
    "{'label': 'ML', 'pattern': 'genetic programming'},\n",
    "{'label': 'ML', 'pattern': 'genetic representation'},\n",
    "{'label': 'ML', 'pattern': 'geographical cluster'},\n",
    "{'label': 'ML', 'pattern': 'gesture description language'},\n",
    "{'label': 'ML', 'pattern': 'geworkbench'},\n",
    "{'label': 'ML', 'pattern': 'glimmer'},\n",
    "{'label': 'ML', 'pattern': 'glottochronology'},\n",
    "{'label': 'ML', 'pattern': 'google brain'},\n",
    "{'label': 'ML', 'pattern': 'google matrix'},\n",
    "{'label': 'ML', 'pattern': 'google nest'},\n",
    "{'label': 'ML', 'pattern': 'google neural machine translation'},\n",
    "{'label': 'ML', 'pattern': 'gpt'},\n",
    "{'label': 'ML', 'pattern': 'gpt-2'},\n",
    "{'label': 'ML', 'pattern': 'gpt-3'},\n",
    "{'label': 'ML', 'pattern': 'gradient boosting'},\n",
    "{'label': 'ML', 'pattern': 'gramian matrix'},\n",
    "{'label': 'ML', 'pattern': 'grammar induction'},\n",
    "{'label': 'ML', 'pattern': 'grammatical evolution'},\n",
    "{'label': 'ML', 'pattern': 'granular computing'},\n",
    "{'label': 'ML', 'pattern': 'graph kernel'},\n",
    "{'label': 'ML', 'pattern': 'grossberg network'},\n",
    "{'label': 'ML', 'pattern': 'group method of data handling'},\n",
    "{'label': 'ML', 'pattern': 'growing self-organizing map'},\n",
    "{'label': 'ML', 'pattern': 'growth function'},\n",
    "{'label': 'ML', 'pattern': 'handwriting recognition'},\n",
    "{'label': 'ML', 'pattern': 'hard sigmoid'},\n",
    "{'label': 'ML', 'pattern': 'hebbian theory'},\n",
    "{'label': 'ML', 'pattern': 'helmholtz machine'},\n",
    "{'label': 'ML', 'pattern': 'hidden markov model'},\n",
    "{'label': 'ML', 'pattern': 'hierarchical classification'},\n",
    "{'label': 'ML', 'pattern': 'hierarchical temporal memory'},\n",
    "{'label': 'ML', 'pattern': 'hinge loss'},\n",
    "{'label': 'ML', 'pattern': 'hopfield network'},\n",
    "{'label': 'ML', 'pattern': 'horovod'},\n",
    "{'label': 'ML', 'pattern': 'huber loss'},\n",
    "{'label': 'ML', 'pattern': 'hybrid kohonen self-organizing map'},\n",
    "{'label': 'ML', 'pattern': 'hybrid neural network'},\n",
    "{'label': 'ML', 'pattern': 'hyper basis function network'},\n",
    "{'label': 'ML', 'pattern': 'hyperneat'},\n",
    "{'label': 'ML', 'pattern': 'hyperparameter'},\n",
    "{'label': 'ML', 'pattern': 'hyperparameter optimization'},\n",
    "{'label': 'ML', 'pattern': 'id3 algorithm'},\n",
    "{'label': 'ML', 'pattern': 'idistance'},\n",
    "{'label': 'ML', 'pattern': 'imagenets'},\n",
    "{'label': 'ML', 'pattern': 'inauthentic text'},\n",
    "{'label': 'ML', 'pattern': 'incremental learning'},\n",
    "{'label': 'ML', 'pattern': 'independent component analysis'},\n",
    "{'label': 'ML', 'pattern': 'induction of regular languages'},\n",
    "{'label': 'ML', 'pattern': 'inductive bias'},\n",
    "{'label': 'ML', 'pattern': 'inductive logic programming'},\n",
    "{'label': 'ML', 'pattern': 'inductive probability'},\n",
    "{'label': 'ML', 'pattern': 'inductive programming'},\n",
    "{'label': 'ML', 'pattern': 'infer.net'},\n",
    "{'label': 'ML', 'pattern': 'inferential theory of learning'},\n",
    "{'label': 'ML', 'pattern': 'influence diagram'},\n",
    "{'label': 'ML', 'pattern': 'infomax'},\n",
    "{'label': 'ML', 'pattern': 'information fuzzy networks'},\n",
    "{'label': 'ML', 'pattern': 'information gain in decision trees'},\n",
    "{'label': 'ML', 'pattern': 'information gain ratio'},\n",
    "{'label': 'ML', 'pattern': 'instance selection'},\n",
    "{'label': 'ML', 'pattern': 'instance-based learning'},\n",
    "{'label': 'ML', 'pattern': 'instantaneously trained neural networks'},\n",
    "{'label': 'ML', 'pattern': 'intel realsense'},\n",
    "{'label': 'ML', 'pattern': 'interacting particle system'},\n",
    "{'label': 'ML', 'pattern': 'interactive activation and competition networks'},\n",
    "{'label': 'ML', 'pattern': 'interactive machine translation'},\n",
    "{'label': 'ML', 'pattern': 'inverted pendulum'},\n",
    "{'label': 'ML', 'pattern': 'ipo underpricing algorithm'},\n",
    "{'label': 'ML', 'pattern': 'ircf360'},\n",
    "{'label': 'ML', 'pattern': 'isolation forest'},\n",
    "{'label': 'ML', 'pattern': 'isotropic position'},\n",
    "{'label': 'ML', 'pattern': 'item response theory'},\n",
    "{'label': 'ML', 'pattern': 'iterative viterbi decoding'},\n",
    "{'label': 'ML', 'pattern': 'java grammatical evolution'},\n",
    "{'label': 'ML', 'pattern': 'jpred'},\n",
    "{'label': 'ML', 'pattern': 'junction tree algorithm'},\n",
    "{'label': 'ML', 'pattern': 'k-nearest neighbors'},\n",
    "{'label': 'ML', 'pattern': 'kalman filter'},\n",
    "{'label': 'ML', 'pattern': 'katzs back-off model'},\n",
    "{'label': 'ML', 'pattern': 'KBpedia series'},\n",
    "{'label': 'ML', 'pattern': 'keras'},\n",
    "{'label': 'ML', 'pattern': 'kernel adaptive filter'},\n",
    "{'label': 'ML', 'pattern': 'kernel density estimation'},\n",
    "{'label': 'ML', 'pattern': 'kernel eigenvoice'},\n",
    "{'label': 'ML', 'pattern': 'kernel embedding of distributions'},\n",
    "{'label': 'ML', 'pattern': 'kernel method'},\n",
    "{'label': 'ML', 'pattern': 'kernel perceptron'},\n",
    "{'label': 'ML', 'pattern': 'kernel principal component analysis'},\n",
    "{'label': 'ML', 'pattern': 'kinect'},\n",
    "{'label': 'ML', 'pattern': 'knowledge distillation'},\n",
    "{'label': 'ML', 'pattern': 'knowledge integration'},\n",
    "{'label': 'ML', 'pattern': 'label propagation algorithm'},\n",
    "{'label': 'ML', 'pattern': 'labeled data'},\n",
    "{'label': 'ML', 'pattern': 'language acquisition device'},\n",
    "{'label': 'ML', 'pattern': 'language model'},\n",
    "{'label': 'ML', 'pattern': 'large margin nearest neighbor'},\n",
    "{'label': 'ML', 'pattern': 'large memory storage and retrieval neural network'},\n",
    "{'label': 'ML', 'pattern': 'latent class model'},\n",
    "{'label': 'ML', 'pattern': 'latent dirichlet allocation'},\n",
    "{'label': 'ML', 'pattern': 'latent semantic analysis'},\n",
    "{'label': 'ML', 'pattern': 'latent variable'},\n",
    "{'label': 'ML', 'pattern': 'latent variable model'},\n",
    "{'label': 'ML', 'pattern': 'lazy learning'},\n",
    "{'label': 'ML', 'pattern': 'leabra'},\n",
    "{'label': 'ML', 'pattern': 'leakage'},\n",
    "{'label': 'ML', 'pattern': 'learnable function class'},\n",
    "{'label': 'ML', 'pattern': 'learning automaton'},\n",
    "{'label': 'ML', 'pattern': 'learning classifier system'},\n",
    "{'label': 'ML', 'pattern': 'learning curve'},\n",
    "{'label': 'ML', 'pattern': 'learning rate'},\n",
    "{'label': 'ML', 'pattern': 'learning rule'},\n",
    "{'label': 'ML', 'pattern': 'learning to rank'},\n",
    "{'label': 'ML', 'pattern': 'learning vector quantization'},\n",
    "{'label': 'ML', 'pattern': 'learning with errors'},\n",
    "{'label': 'ML', 'pattern': 'least-squares support-vector machine'},\n",
    "{'label': 'ML', 'pattern': 'leave-one-out error'},\n",
    "{'label': 'ML', 'pattern': 'leela chess zero'},\n",
    "{'label': 'ML', 'pattern': 'leela zero'},\n",
    "{'label': 'ML', 'pattern': 'lenet'},\n",
    "{'label': 'ML', 'pattern': 'lernmatrix'},\n",
    "{'label': 'ML', 'pattern': 'life-time of correlation'},\n",
    "{'label': 'ML', 'pattern': 'lightgbm'},\n",
    "{'label': 'ML', 'pattern': 'linde–buzo–gray algorithm'},\n",
    "{'label': 'ML', 'pattern': 'linear classifier'},\n",
    "{'label': 'ML', 'pattern': 'linear discriminant analysis'},\n",
    "{'label': 'ML', 'pattern': 'linear genetic programming'},\n",
    "{'label': 'ML', 'pattern': 'linear predictor function'},\n",
    "{'label': 'ML', 'pattern': 'linear separability'},\n",
    "{'label': 'ML', 'pattern': 'liquid state machine'},\n",
    "{'label': 'ML', 'pattern': 'list of datasets for machine-learning research'},\n",
    "{'label': 'ML', 'pattern': 'local case-control sampling'},\n",
    "{'label': 'ML', 'pattern': 'local independence'},\n",
    "{'label': 'ML', 'pattern': 'local outlier factor'},\n",
    "{'label': 'ML', 'pattern': 'local tangent space alignment'},\n",
    "{'label': 'ML', 'pattern': 'locality-sensitive hashing'},\n",
    "{'label': 'ML', 'pattern': 'log-linear model'},\n",
    "{'label': 'ML', 'pattern': 'logic learning machine'},\n",
    "{'label': 'ML', 'pattern': 'logitboost'},\n",
    "{'label': 'ML', 'pattern': 'long short-term memory'},\n",
    "{'label': 'ML', 'pattern': 'loss function'},\n",
    "{'label': 'ML', 'pattern': 'loss functions for classification'},\n",
    "{'label': 'ML', 'pattern': 'low-rank approximation'},\n",
    "{'label': 'ML', 'pattern': 'low-rank matrix approximations'},\n",
    "{'label': 'ML', 'pattern': 'lpboost'},\n",
    "{'label': 'ML', 'pattern': 'm-theory'},\n",
    "{'label': 'ML', 'pattern': 'machine learning'},\n",
    "{'label': 'ML', 'pattern': 'machine_learning'},            \n",
    "{'label': 'ML', 'pattern': 'manifold alignment'},\n",
    "{'label': 'ML', 'pattern': 'manifold regularization'},\n",
    "{'label': 'ML', 'pattern': 'margin classifier'},\n",
    "{'label': 'ML', 'pattern': 'margin-infused relaxed algorithm'},\n",
    "{'label': 'ML', 'pattern': 'markov blanket'},\n",
    "{'label': 'ML', 'pattern': 'markov chain'},\n",
    "{'label': 'ML', 'pattern': 'markov chain central limit theorem'},\n",
    "{'label': 'ML', 'pattern': 'markov chain geostatistics'},\n",
    "{'label': 'ML', 'pattern': 'markov chain monte carlo'},\n",
    "{'label': 'ML', 'pattern': 'markov information source'},\n",
    "{'label': 'ML', 'pattern': 'markov model'},\n",
    "{'label': 'ML', 'pattern': 'markov partition'},\n",
    "{'label': 'ML', 'pattern': 'markov property'},\n",
    "{'label': 'ML', 'pattern': 'markov switching multifractal'},\n",
    "{'label': 'ML', 'pattern': 'markovian discrimination'},\n",
    "{'label': 'ML', 'pattern': 'matchbox educable noughts and crosses engine'},\n",
    "{'label': 'ML', 'pattern': 'matrix regularization'},\n",
    "{'label': 'ML', 'pattern': 'matthews correlation coefficient'},\n",
    "{'label': 'ML', 'pattern': 'maximum-entropy markov model'},\n",
    "{'label': 'ML', 'pattern': 'mean squared error'},\n",
    "{'label': 'ML', 'pattern': 'mean squared prediction error'},\n",
    "{'label': 'ML', 'pattern': 'measurement invariance'},\n",
    "{'label': 'ML', 'pattern': 'medoid'},\n",
    "{'label': 'ML', 'pattern': 'megahal'},\n",
    "{'label': 'ML', 'pattern': 'melomics'},\n",
    "{'label': 'ML', 'pattern': 'memetic algorithm'},\n",
    "{'label': 'ML', 'pattern': 'memtransistor'},\n",
    "{'label': 'ML', 'pattern': 'meta learning'},\n",
    "{'label': 'ML', 'pattern': 'meta-optimization'},\n",
    "{'label': 'ML', 'pattern': 'microsoft cognitive toolkit'},\n",
    "{'label': 'ML', 'pattern': 'minimum population search'},\n",
    "{'label': 'ML', 'pattern': 'minimum redundancy feature selection'},\n",
    "{'label': 'ML', 'pattern': 'mixture model'},\n",
    "{'label': 'ML', 'pattern': 'mixture of experts'},\n",
    "{'label': 'ML', 'pattern': 'ml.net'},\n",
    "{'label': 'ML', 'pattern': 'mlops'},\n",
    "{'label': 'ML', 'pattern': 'model-free'},\n",
    "{'label': 'ML', 'pattern': 'models of dna evolution'},\n",
    "{'label': 'ML', 'pattern': 'modes of variation'},\n",
    "{'label': 'ML', 'pattern': 'modular neural network'},\n",
    "{'label': 'ML', 'pattern': 'moea framework'},\n",
    "{'label': 'ML', 'pattern': 'mokken scale'},\n",
    "{'label': 'ML', 'pattern': 'moneybee'},\n",
    "{'label': 'ML', 'pattern': 'moral graph'},\n",
    "{'label': 'ML', 'pattern': 'mountain car problem'},\n",
    "{'label': 'ML', 'pattern': 'multi expression programming'},\n",
    "{'label': 'ML', 'pattern': 'multi-agent learning'},\n",
    "{'label': 'ML', 'pattern': 'multi-armed bandit'},\n",
    "{'label': 'ML', 'pattern': 'multi-label classification'},\n",
    "{'label': 'ML', 'pattern': 'multi-objective reinforcement learning'},\n",
    "{'label': 'ML', 'pattern': 'multi-surface method'},\n",
    "{'label': 'ML', 'pattern': 'multi-task learning'},\n",
    "{'label': 'ML', 'pattern': 'multiclass classification'},\n",
    "{'label': 'ML', 'pattern': 'multidimensional analysis'},\n",
    "{'label': 'ML', 'pattern': 'multidimensional scaling'},\n",
    "{'label': 'ML', 'pattern': 'multifactor dimensionality reduction'},\n",
    "{'label': 'ML', 'pattern': 'multilayer perceptron'},\n",
    "{'label': 'ML', 'pattern': 'multilinear principal component analysis'},\n",
    "{'label': 'ML', 'pattern': 'multilinear subspace learning'},\n",
    "{'label': 'ML', 'pattern': 'multimodal learning'},\n",
    "{'label': 'ML', 'pattern': 'multimodal sentiment analysis'},\n",
    "{'label': 'ML', 'pattern': 'multinomial logistic regression'},\n",
    "{'label': 'ML', 'pattern': 'multiple correspondence analysis'},\n",
    "{'label': 'ML', 'pattern': 'multiple discriminant analysis'},\n",
    "{'label': 'ML', 'pattern': 'multiple discriminant analysis'},\n",
    "{'label': 'ML', 'pattern': 'multiple instance learning'},\n",
    "{'label': 'ML', 'pattern': 'multiple kernel learning'},\n",
    "{'label': 'ML', 'pattern': 'multiple sequence alignment'},\n",
    "{'label': 'ML', 'pattern': 'multiple-instance learning'},\n",
    "{'label': 'ML', 'pattern': 'multiplicative weight update method'},\n",
    "{'label': 'ML', 'pattern': 'multispectral pattern recognition'},\n",
    "{'label': 'ML', 'pattern': 'multitask optimization'},\n",
    "{'label': 'ML', 'pattern': 'multivariate adaptive regression spline'},\n",
    "{'label': 'ML', 'pattern': 'naive bayes classifier'},\n",
    "{'label': 'ML', 'pattern': 'native-language identification'},\n",
    "{'label': 'ML', 'pattern': 'natural evolution strategy'},\n",
    "{'label': 'ML', 'pattern': 'natural language toolkit'},\n",
    "{'label': 'ML', 'pattern': 'nature machine intelligence'},\n",
    "{'label': 'ML', 'pattern': 'nearest centroid classifier'},\n",
    "{'label': 'ML', 'pattern': 'nearest neighbor search'},\n",
    "{'label': 'ML', 'pattern': 'neocognitron'},\n",
    "{'label': 'ML', 'pattern': 'netomi'},\n",
    "{'label': 'ML', 'pattern': 'nettalk'},\n",
    "{'label': 'ML', 'pattern': 'neural cryptography'},\n",
    "{'label': 'ML', 'pattern': 'neural designer'},\n",
    "{'label': 'ML', 'pattern': 'neural gas'},\n",
    "{'label': 'ML', 'pattern': 'neural modeling fields'},\n",
    "{'label': 'ML', 'pattern': 'neural network gaussian process'},\n",
    "{'label': 'ML', 'pattern': 'neural network intelligence'},\n",
    "{'label': 'ML', 'pattern': 'neural network software'},\n",
    "{'label': 'ML', 'pattern': 'neural network synchronization protocol'},\n",
    "{'label': 'ML', 'pattern': 'neural networks'},\n",
    "{'label': 'ML', 'pattern': 'neural style transfer'},\n",
    "{'label': 'ML', 'pattern': 'neural tangent kernel'},\n",
    "{'label': 'ML', 'pattern': 'neural turing machine'},\n",
    "{'label': 'ML', 'pattern': 'neuroevolution'},\n",
    "{'label': 'ML', 'pattern': 'neuroevolution of augmenting topologies'},\n",
    "{'label': 'ML', 'pattern': 'ni1000'},\n",
    "{'label': 'ML', 'pattern': 'niki.ai'},\n",
    "{'label': 'ML', 'pattern': 'node2vec'},\n",
    "{'label': 'ML', 'pattern': 'noisy channel model'},\n",
    "{'label': 'ML', 'pattern': 'noisy text analytics'},\n",
    "{'label': 'ML', 'pattern': 'non-negative matrix factorization'},\n",
    "{'label': 'ML', 'pattern': 'nonlinear dimensionality reduction'},\n",
    "{'label': 'ML', 'pattern': 'normal discriminant analysis'},\n",
    "{'label': 'ML', 'pattern': 'novelty detection'},\n",
    "{'label': 'ML', 'pattern': 'nuisance variable'},\n",
    "{'label': 'ML', 'pattern': 'nvdla'},\n",
    "{'label': 'ML', 'pattern': 'object co-segmentation'},\n",
    "{'label': 'ML', 'pattern': 'occam learning'},\n",
    "{'label': 'ML', 'pattern': 'offline learning'},\n",
    "{'label': 'ML', 'pattern': 'ojas rule'},\n",
    "{'label': 'ML', 'pattern': 'one-class classification'},\n",
    "{'label': 'ML', 'pattern': 'one-shot learning'},\n",
    "{'label': 'ML', 'pattern': 'online machine learning'},\n",
    "{'label': 'ML', 'pattern': 'onnx'},\n",
    "{'label': 'ML', 'pattern': 'ontology learning'},\n",
    "{'label': 'ML', 'pattern': 'openai api'},\n",
    "{'label': 'ML', 'pattern': 'openai five'},\n",
    "{'label': 'ML', 'pattern': 'opennn'},\n",
    "{'label': 'ML', 'pattern': 'openvino'},\n",
    "{'label': 'ML', 'pattern': 'operational taxonomic unit'},\n",
    "{'label': 'ML', 'pattern': 'optical character recognition'},\n",
    "{'label': 'ML', 'pattern': 'optical neural network'},\n",
    "{'label': 'ML', 'pattern': 'optimal discriminant analysis and classification tree analysis'},\n",
    "{'label': 'ML', 'pattern': 'oscillatory neural network'},\n",
    "{'label': 'ML', 'pattern': 'out-of-bag error'},\n",
    "{'label': 'ML', 'pattern': 'outline of machine learning'},\n",
    "{'label': 'ML', 'pattern': 'overfitting'},\n",
    "{'label': 'ML', 'pattern': 'pachinko allocation'},\n",
    "{'label': 'ML', 'pattern': 'pagerank'},\n",
    "{'label': 'ML', 'pattern': 'paraphrasing'},\n",
    "{'label': 'ML', 'pattern': 'parity benchmark'},\n",
    "{'label': 'ML', 'pattern': 'parity learning'},\n",
    "{'label': 'ML', 'pattern': 'part-of-speech tagging'},\n",
    "{'label': 'ML', 'pattern': 'partial least squares regression'},\n",
    "{'label': 'ML', 'pattern': 'particle swarm optimization'},\n",
    "{'label': 'ML', 'pattern': 'path dependence'},\n",
    "{'label': 'ML', 'pattern': 'pattern language'},\n",
    "{'label': 'ML', 'pattern': 'pattern recognition'},\n",
    "{'label': 'ML', 'pattern': 'perceptron'},\n",
    "{'label': 'ML', 'pattern': 'physical neural network'},\n",
    "{'label': 'ML', 'pattern': 'plate notation'},\n",
    "{'label': 'ML', 'pattern': 'polynomial kernel'},\n",
    "{'label': 'ML', 'pattern': 'pop music automation'},\n",
    "{'label': 'ML', 'pattern': 'population process'},\n",
    "{'label': 'ML', 'pattern': 'portable format for analytics'},\n",
    "{'label': 'ML', 'pattern': 'predictive learning'},\n",
    "{'label': 'ML', 'pattern': 'predictive model markup language'},\n",
    "{'label': 'ML', 'pattern': 'predictive state representation'},\n",
    "{'label': 'ML', 'pattern': 'preference learning'},\n",
    "{'label': 'ML', 'pattern': 'preference regression'},\n",
    "{'label': 'ML', 'pattern': 'prefrontal cortex basal ganglia working memory'},\n",
    "{'label': 'ML', 'pattern': 'principal component analysis'},\n",
    "{'label': 'ML', 'pattern': 'prior knowledge for pattern recognition'},\n",
    "{'label': 'ML', 'pattern': 'proactive learning'},\n",
    "{'label': 'ML', 'pattern': 'proaftn'},\n",
    "{'label': 'ML', 'pattern': 'probabilistic context-free grammar'},\n",
    "{'label': 'ML', 'pattern': 'probabilistic latent semantic analysis'},\n",
    "{'label': 'ML', 'pattern': 'probabilistic neural network'},\n",
    "{'label': 'ML', 'pattern': 'probability matching'},\n",
    "{'label': 'ML', 'pattern': 'probably approximately correct learning'},\n",
    "{'label': 'ML', 'pattern': 'probit model'},\n",
    "{'label': 'ML', 'pattern': 'product of experts'},\n",
    "{'label': 'ML', 'pattern': 'progol'},\n",
    "{'label': 'ML', 'pattern': 'programming by example'},\n",
    "{'label': 'ML', 'pattern': 'promoter based genetic algorithm'},\n",
    "{'label': 'ML', 'pattern': 'proper generalized decomposition'},\n",
    "{'label': 'ML', 'pattern': 'prototype methods'},\n",
    "{'label': 'ML', 'pattern': 'proximal gradient method'},\n",
    "{'label': 'ML', 'pattern': 'pulse-coupled networks'},\n",
    "{'label': 'ML', 'pattern': 'pvlv'},\n",
    "{'label': 'ML', 'pattern': 'q-learning'},\n",
    "{'label': 'ML', 'pattern': 'quadratic classifier'},\n",
    "{'label': 'ML', 'pattern': 'quadratic unconstrained binary optimization'},\n",
    "{'label': 'ML', 'pattern': 'quantum machine learning'},\n",
    "{'label': 'ML', 'pattern': 'quantum markov chain'},\n",
    "{'label': 'ML', 'pattern': 'quantum neural network'},\n",
    "{'label': 'ML', 'pattern': 'query-level feature'},\n",
    "{'label': 'ML', 'pattern': 'question answering'},\n",
    "{'label': 'ML', 'pattern': 'queueing theory'},\n",
    "{'label': 'ML', 'pattern': 'quickprop'},\n",
    "{'label': 'ML', 'pattern': 'rademacher complexity'},\n",
    "{'label': 'ML', 'pattern': 'radial basis function'},\n",
    "{'label': 'ML', 'pattern': 'radial basis function kernel'},\n",
    "{'label': 'ML', 'pattern': 'radial basis function network'},\n",
    "{'label': 'ML', 'pattern': 'ramnets'},\n",
    "{'label': 'ML', 'pattern': 'random forest'},\n",
    "{'label': 'ML', 'pattern': 'random indexing'},\n",
    "{'label': 'ML', 'pattern': 'random neural network'},\n",
    "{'label': 'ML', 'pattern': 'random projection'},\n",
    "{'label': 'ML', 'pattern': 'random subspace method'},\n",
    "{'label': 'ML', 'pattern': 'randomized weighted majority algorithm'},\n",
    "{'label': 'ML', 'pattern': 'ranking svm'},\n",
    "{'label': 'ML', 'pattern': 'reasoning system'},\n",
    "{'label': 'ML', 'pattern': 'rectifier'},\n",
    "{'label': 'ML', 'pattern': 'recurrent neural network'},\n",
    "{'label': 'ML', 'pattern': 'recursive neural network'},\n",
    "{'label': 'ML', 'pattern': 'region based convolutional neural networks'},\n",
    "{'label': 'ML', 'pattern': 'reinforcement learning'},\n",
    "{'label': 'ML', 'pattern': 'relation network'},\n",
    "{'label': 'ML', 'pattern': 'relational data mining'},\n",
    "{'label': 'ML', 'pattern': 'relationship square'},\n",
    "{'label': 'ML', 'pattern': 'relevance vector machine'},\n",
    "{'label': 'ML', 'pattern': 'representer theorem'},\n",
    "{'label': 'ML', 'pattern': 'reservoir computing'},\n",
    "{'label': 'ML', 'pattern': 'residual neural network'},\n",
    "{'label': 'ML', 'pattern': 'restricted boltzmann machine'},\n",
    "{'label': 'ML', 'pattern': 'revoscalepy'},\n",
    "{'label': 'ML', 'pattern': 'revoscaler'},\n",
    "{'label': 'ML', 'pattern': 'reward-based selection'},\n",
    "{'label': 'ML', 'pattern': 'right to explanation'},\n",
    "{'label': 'ML', 'pattern': 'rnn'},\n",
    "{'label': 'ML', 'pattern': 'robot learning'},\n",
    "{'label': 'ML', 'pattern': 'robotic process automation'},\n",
    "{'label': 'ML', 'pattern': 'robust principal component analysis'},\n",
    "{'label': 'ML', 'pattern': 'rprop'},\n",
    "{'label': 'ML', 'pattern': 'rule induction'},\n",
    "{'label': 'ML', 'pattern': 'rule-based machine learning'},\n",
    "{'label': 'ML', 'pattern': 'rules extraction system family'},\n",
    "{'label': 'ML', 'pattern': 'sammon mapping'},\n",
    "{'label': 'ML', 'pattern': 'sample complexity'},\n",
    "{'label': 'ML', 'pattern': 'sample exclusion dimension'},\n",
    "{'label': 'ML', 'pattern': 'santa fe trail problem'},\n",
    "{'label': 'ML', 'pattern': 'scale-invariant feature operator'},\n",
    "{'label': 'ML', 'pattern': 'scikit-multiflow'},\n",
    "{'label': 'ML', 'pattern': 'self-organizing map'},\n",
    "{'label': 'ML', 'pattern': 'semantic analysis'},\n",
    "{'label': 'ML', 'pattern': 'semantic folding'},\n",
    "{'label': 'ML', 'pattern': 'semantic mapping'},\n",
    "{'label': 'ML', 'pattern': 'semantic neural network'},\n",
    "{'label': 'ML', 'pattern': 'semi-supervised learning'},\n",
    "{'label': 'ML', 'pattern': 'semidefinite embedding'},\n",
    "{'label': 'ML', 'pattern': 'sense networks'},\n",
    "{'label': 'ML', 'pattern': 'sentence embedding'},\n",
    "{'label': 'ML', 'pattern': 'seq2seq'},\n",
    "{'label': 'ML', 'pattern': 'sequence labeling'},\n",
    "{'label': 'ML', 'pattern': 'sequential minimal optimization'},\n",
    "{'label': 'ML', 'pattern': 'shattered set'},\n",
    "{'label': 'ML', 'pattern': 'siamese neural network'},\n",
    "{'label': 'ML', 'pattern': 'sigmoid function'},\n",
    "{'label': 'ML', 'pattern': 'similarity learning'},\n",
    "{'label': 'ML', 'pattern': 'simultaneous localization and mapping'},\n",
    "{'label': 'ML', 'pattern': 'sinkov statistic'},\n",
    "{'label': 'ML', 'pattern': 'skill chaining'},\n",
    "{'label': 'ML', 'pattern': 'sliced inverse regression'},\n",
    "{'label': 'ML', 'pattern': 'soboleva modified hyperbolic tangent'},\n",
    "{'label': 'ML', 'pattern': 'soft output viterbi algorithm'},\n",
    "{'label': 'ML', 'pattern': 'softmax function'},\n",
    "{'label': 'ML', 'pattern': 'solomonoffs theory of inductive inference'},\n",
    "{'label': 'ML', 'pattern': 'sparse dictionary learning'},\n",
    "{'label': 'ML', 'pattern': 'sparse pca'},\n",
    "{'label': 'ML', 'pattern': 'speech recognition'},\n",
    "{'label': 'ML', 'pattern': 'spike-and-slab regression'},\n",
    "{'label': 'ML', 'pattern': 'spiking neural network'},\n",
    "{'label': 'ML', 'pattern': 'spiral optimization algorithm'},\n",
    "{'label': 'ML', 'pattern': 'squeezenet'},\n",
    "{'label': 'ML', 'pattern': 'state–action–reward–state–action'},\n",
    "{'label': 'ML', 'pattern': 'statistical classification'},\n",
    "{'label': 'ML', 'pattern': 'statistical learning theory'},\n",
    "{'label': 'ML', 'pattern': 'statistical machine translation'},\n",
    "{'label': 'ML', 'pattern': 'statistical parsing'},\n",
    "{'label': 'ML', 'pattern': 'statistical relational learning'},\n",
    "{'label': 'ML', 'pattern': 'statistical semantics'},\n",
    "{'label': 'ML', 'pattern': 'stochastic block model'},\n",
    "{'label': 'ML', 'pattern': 'stochastic cellular automaton'},\n",
    "{'label': 'ML', 'pattern': 'stochastic gradient descent'},\n",
    "{'label': 'ML', 'pattern': 'stochastic grammar'},\n",
    "{'label': 'ML', 'pattern': 'stochastic matrix'},\n",
    "{'label': 'ML', 'pattern': 'stochastic neural analog reinforcement calculator'},\n",
    "{'label': 'ML', 'pattern': 'stochastic neural network'},\n",
    "{'label': 'ML', 'pattern': 'stress majorization'},\n",
    "{'label': 'ML', 'pattern': 'string kernel'},\n",
    "{'label': 'ML', 'pattern': 'structural equation modeling'},\n",
    "{'label': 'ML', 'pattern': 'structural risk minimization'},\n",
    "{'label': 'ML', 'pattern': 'structured knn'},\n",
    "{'label': 'ML', 'pattern': 'structured prediction'},\n",
    "{'label': 'ML', 'pattern': 'structured sparsity regularization'},\n",
    "{'label': 'ML', 'pattern': 'structured support vector machine'},\n",
    "{'label': 'ML', 'pattern': 'stylegan'},\n",
    "{'label': 'ML', 'pattern': 'subclass reachability'},\n",
    "{'label': 'ML', 'pattern': 'sufficient dimension reduction'},\n",
    "{'label': 'ML', 'pattern': 'sukhotins algorithm'},\n",
    "{'label': 'ML', 'pattern': 'sum of absolute differences'},\n",
    "{'label': 'ML', 'pattern': 'sum of absolute transformed differences'},\n",
    "{'label': 'ML', 'pattern': 'supervised learning'},\n",
    "{'label': 'ML', 'pattern': 'support vector machine'},\n",
    "{'label': 'ML', 'pattern': 'swish function'},\n",
    "{'label': 'ML', 'pattern': 'switching kalman filter'},\n",
    "{'label': 'ML', 'pattern': 'symbolic regression'},\n",
    "{'label': 'ML', 'pattern': 'synaptic transistor'},\n",
    "{'label': 'ML', 'pattern': 'synaptic weight'},\n",
    "{'label': 'ML', 'pattern': 'synchronous context-free grammar'},\n",
    "{'label': 'ML', 'pattern': 'syntactic pattern recognition'},\n",
    "{'label': 'ML', 'pattern': 't-distributed stochastic neighbor embedding'},\n",
    "{'label': 'ML', 'pattern': 'taguchi loss function'},\n",
    "{'label': 'ML', 'pattern': 'tastedive'},\n",
    "{'label': 'ML', 'pattern': 'td-gammon'},\n",
    "{'label': 'ML', 'pattern': 'teaching dimension'},\n",
    "{'label': 'ML', 'pattern': 'temporal difference learning'},\n",
    "{'label': 'ML', 'pattern': 'tensor product network'},\n",
    "{'label': 'ML', 'pattern': 'tensor sketch'},\n",
    "{'label': 'ML', 'pattern': 'tensorflow'},\n",
    "{'label': 'ML', 'pattern': 'text mining'},\n",
    "{'label': 'ML', 'pattern': 'textual case-based reasoning'},\n",
    "{'label': 'ML', 'pattern': 'tf–idf'},\n",
    "{'label': 'ML', 'pattern': 'the emotion machine'},\n",
    "{'label': 'ML', 'pattern': 'the master algorithm'},\n",
    "{'label': 'ML', 'pattern': 'theano'},\n",
    "{'label': 'ML', 'pattern': 'theory of conjoint measurement'},\n",
    "{'label': 'ML', 'pattern': 'thurstonian model'},\n",
    "{'label': 'ML', 'pattern': 'time aware long short-term memory'},\n",
    "{'label': 'ML', 'pattern': 'time delay neural network'},\n",
    "{'label': 'ML', 'pattern': 'time series'},\n",
    "{'label': 'ML', 'pattern': 'timeline of machine learning'},\n",
    "{'label': 'ML', 'pattern': 'topic model'},\n",
    "{'label': 'ML', 'pattern': 'training, validation, and test sets'},\n",
    "{'label': 'ML', 'pattern': 'transduction'},\n",
    "{'label': 'ML', 'pattern': 'transfer learning'},\n",
    "{'label': 'ML', 'pattern': 'transformer'},\n",
    "{'label': 'ML', 'pattern': 'trigram tagger'},\n",
    "{'label': 'ML', 'pattern': 'triplet loss'},\n",
    "{'label': 'ML', 'pattern': 'tsetlin machine'},\n",
    "{'label': 'ML', 'pattern': 'tucker decomposition'},\n",
    "{'label': 'ML', 'pattern': 'types of artificial neural networks'},\n",
    "{'label': 'ML', 'pattern': 'u-matrix'},\n",
    "{'label': 'ML', 'pattern': 'u-net'},\n",
    "{'label': 'ML', 'pattern': 'ugly duckling theorem'},\n",
    "{'label': 'ML', 'pattern': 'uncertain data'},\n",
    "{'label': 'ML', 'pattern': 'under-fitting'},\n",
    "{'label': 'ML', 'pattern': 'underfitting'},\n",
    "{'label': 'ML', 'pattern': 'uniform convergence in probability'},\n",
    "{'label': 'ML', 'pattern': 'unique negative dimension'},\n",
    "{'label': 'ML', 'pattern': 'universal approximation theorem'},\n",
    "{'label': 'ML', 'pattern': 'universal portfolio algorithm'},\n",
    "{'label': 'ML', 'pattern': 'unsupervised learning'},\n",
    "{'label': 'ML', 'pattern': 'user behavior analytics'},\n",
    "{'label': 'ML', 'pattern': 'validation set'},\n",
    "{'label': 'ML', 'pattern': 'vanishing gradient problem'},\n",
    "{'label': 'ML', 'pattern': 'vapnik–chervonenkis dimension'},\n",
    "{'label': 'ML', 'pattern': 'vapnik–chervonenkis theory'},\n",
    "{'label': 'ML', 'pattern': 'variable kernel density estimation'},\n",
    "{'label': 'ML', 'pattern': 'variable-order bayesian network'},\n",
    "{'label': 'ML', 'pattern': 'variable-order markov model'},\n",
    "{'label': 'ML', 'pattern': 'variational message passing'},\n",
    "{'label': 'ML', 'pattern': 'vector quantization'},\n",
    "{'label': 'ML', 'pattern': 'version space learning'},\n",
    "{'label': 'ML', 'pattern': 'visual temporal attention'},\n",
    "{'label': 'ML', 'pattern': 'viterbi algorithm'},\n",
    "{'label': 'ML', 'pattern': 'waca clustering algorithm'},\n",
    "{'label': 'ML', 'pattern': 'waifu2x'},\n",
    "{'label': 'ML', 'pattern': 'wake-sleep algorithm'},\n",
    "{'label': 'ML', 'pattern': 'wavenet'},\n",
    "{'label': 'ML', 'pattern': 'weak supervision'},\n",
    "{'label': 'ML', 'pattern': 'weighted majority algorithm'},\n",
    "{'label': 'ML', 'pattern': 'whitening transformation'},\n",
    "{'label': 'ML', 'pattern': 'witness set'},\n",
    "{'label': 'ML', 'pattern': 'word embedding'},\n",
    "{'label': 'ML', 'pattern': 'word2vec'},\n",
    "{'label': 'ML', 'pattern': 'writer invariant'},\n",
    "{'label': 'ML', 'pattern': 'zero-shot learning'}]\n",
    "\n",
    "ruler.add_patterns(patterns)\n",
    "nlp.add_pipe(ruler, before='ner')\n",
    "\n",
    "nlp.to_disk(output_dir)\n",
    "print('Saved model to: ', output_dir) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rule-based matcher provides another step in the processing pipeline, so it can be readily combined with the existing NER recognizer. After completing the routine above, we are now able to invoke our new model which combines the existing <code>en_core_web_sm</code> model and the new ruler pipeline step into our new model, <code>en_core_ml</code>, as the code below shows. The following code takes our new model and uses it to generate a listing of the tags found in our input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORG the Cooking with Python\n",
      "ML KBpedia series\n",
      "LOC Part VI\n",
      "CARDINAL seven\n",
      "ML machine learning\n",
      "CARDINAL 11\n",
      "CARDINAL four\n",
      "GPE roadmap\n",
      "GPE KBpedia\n",
      "GPE KBpedia\n",
      "CARDINAL One\n",
      "ORG KBpedia's\n",
      "CARDINAL three\n",
      "ML machine learning\n",
      "ORDINAL First\n",
      "CARDINAL about 56,000\n",
      "GPE KBpedia\n",
      "ML machine learning\n",
      "GPE KBpedia\n",
      "CARDINAL tens of millions\n",
      "CARDINAL tens of thousands\n",
      "GPE KBpedia\n",
      "ORDINAL Second\n",
      "PERCENT 80%\n",
      "GPE KBpedia\n",
      "ML word embedding\n",
      "GPE KBpedia\n",
      "GPE KBpedia\n",
      "ML word embedding\n",
      "ORDINAL third\n",
      "ML machine learning\n",
      "ML deep learning\n",
      "ORDINAL first\n",
      "ML machine learning\n",
      "ML neural networks\n",
      "ML neural networks\n",
      "ML CNN\n",
      "ML neural networks\n",
      "ORG RNN\n",
      "PERSON Graphs\n",
      "ML deep learning\n",
      "DATE the past five years\n",
      "CARDINAL eleven\n",
      "ORDINAL first\n",
      "ORG Python\n",
      "ML machine learning\n",
      "PERSON PyTorch\n",
      "CARDINAL four\n",
      "ORG NLP\n",
      "ML deep learning\n",
      "CARDINAL two\n",
      "CARDINAL four\n",
      "GPE Clojure\n",
      "GPE KBpedia\n",
      "GPE AI\n",
      "CARDINAL one\n",
      "ORG Python\n",
      "ORDINAL first\n",
      "ML machine learning\n",
      "PERSON Ronald Fisher\n",
      "DATE the 1930s\n",
      "GPE Iris\n",
      "DATE today\n",
      "ML linear discriminant analysis\n",
      "CARDINAL dozens\n",
      "ML machine learning\n",
      "CARDINAL 1\n",
      "CARDINAL one\n",
      "PERSON ML\n",
      "ML machine learning\n",
      "NORP DL\n",
      "ML deep learning\n",
      "ORG Python\n",
      "PERSON S. Chen\n",
      "WORK_OF_ART Machine Learning Algorithms For Beginners with Code Examples in Python\n",
      "DATE June 2020\n",
      "CARDINAL one\n",
      "ML deep learning\n",
      "ML machine learning\n",
      "ML supervised learning\n",
      "ML unsupervised learning\n",
      "ML reinforcement learning\n",
      "ML machine learning\n",
      "PRODUCT Graphs\n",
      "ML machine learning\n",
      "QUANTITY 5 yr\n",
      "ML machine learning\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "#import en_core_web_sm\n",
    "import random\n",
    "from spacy.util import minibatch, compounding\n",
    " \n",
    "#nlp = en_core_web_sm.load()\n",
    "\n",
    "model = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\en_core_ml'\n",
    " \n",
    "nlp = spacy.load(model)\n",
    "#nlp.add_pipe(ruler)\n",
    "\n",
    "ner=nlp.get_pipe('ner')\n",
    "\n",
    "text = \"\"\"With this installment of the Cooking with Python and KBpedia series we move into Part VI of seven parts, a part with the bulk of the analytical and machine learning (that is, \"data science\") discussion, and the last part where significant code is developed and documented. At the conclusion of this part, which itself has 11 installments, we have four installments to wrap up the series and provide a consistent roadmap to the entire project.\n",
    "      Knowledge graphs are unique information artifacts, and KBpedia is further unique in terms of its consistent and logical construction as well as its incorporation of significant text content via Wikipedia pages. These characteristics provide unique value for KBpedia, but it is also a combination not duplicated anywhere else in the data science ecosystem. One of the objectives, therefore, of this part of our CWPK series is the creation of some baseline knowledge representations useful to data science aims that capture these unique characteristics.\n",
    "      KBpedia's (or any knowledge graph constructed in a similar manner) combination of characteristics make it a powerful resource in three areas of data science and machine learning. First, the nearly universal scope and degree of topic coverage with about 56,000 concepts, logically organized into typologies with a high degree of disjointedness, means that accurate 'slices' or training sets may be extracted from KBpedia nearly instantaneously. Labeled training sets are one of the most time consuming and expensive activities in doing supervised machine learning. We can extract these nearly for free from KBpedia. Further, with its links to tens of millions of entities in its mapped knowledge bases such as Wikidata, literally tens of thousands of conceptual entities in KBpedia can be the retrieval points to nucleate training sets for fine-grained entity recognition.\n",
    "      Second, 80% of KBpedia's concepts are mapped to Wikipedia articles. While many Wikipedia-based word embedding models exist, the ones in KBpedia are logically categorized and have rough equivalence in terms of scope and prominence, hopefully providing cleaner topic 'signals'. To probe these assertions, we will create a unique KBpedia-based word embedding corpus that also leverages labels for items of structural importance, such as typology membership. We will use this corpus in many of our tests and as a general focus in our training sets.\n",
    "      And, third, perhaps the most important area, knowledge graphs offer unique structures and challenges for machine learning, especially innovations in geometric, heterogeneous methods for deep learning. The first generation of deep machine learning was designed for grid-patterned data and matrices through approaches such as deep neural networks, convolutional neural networks (CNN ), or recurrent neural networks (RNN). The 'deep' appelation comes from having multiple calculated, intermediate layers of transformations between the grid inputs and outputs for the model. Graphs, on the other hand, are heterogeneous between nodes and edges. They may be directed (subsumptive) in nature. And, for knowledge graphs, they have much labeling and annotation, including varying degrees of attribute completeness. Language embedding, itself often a product of deep learning, enables the efficient incorporation of text. It is only in the past five years that concerted attention has been devoted to better capturing this feature richness for knowledge graphs.\n",
    "      The eleven installments in this part will look in more depth at networks and graphs, focus on how to create training sets and embeddings for the learners, discuss some natural language packages and uses, and then look in depth at 'standard' machine learners and deep learners. We will install the first generation of deep graph learners and then explore some on the cutting edge. We will test many use cases, but will also try to invoke classifiers across this spectrum so that we can draw some general conclusions.\n",
    "      The material below introduces and tees up these topics. We describe leading Python packages for data science, and how we have architected our own approach, We have picked a particular Python machine learning framework, PyTorch, to which we will then tie four different NLP and deep learning libraries. We devote two installments each to these four libraries. The use cases we document across these installments are in addition to the existing ones we have in Clojure posted online.\n",
    "      So, we think we have an interesting suite of benefits to cover in this part, some arising from being based on KBpedia and some arising from the nature of knowledge graphs. On the other hand, due to the relative immaturity of the field, we are still actively learning and innovating around the juncture of AI and knowledge graphs. Thus, one of the reasons we emphasize Python 'ecosystems' and 'frameworks' in this part is to be better prepared to incorporate those innovations and learnings to come.\n",
    "      Background\n",
    "      One of the first prototypes of machine learning comes from the statistician Ronald Fisher in the 1930s regarding how to classify Iris species based on the attributes of their flowers. It was a multivariate data example using the method we today call linear discriminant analysis. This classic example is still taught. But many dozens of new algorithms and combined approaches have joined the machine learning field since then.\n",
    "      Figure 1 below is one way to characterize the field, with ML standing for machine learning and DL for deep learning, with this one oriented to sub-fields in which some Python package already exists:\n",
    "      Machine Learning Landscape\n",
    "      Figure 1: Machine Learning Landscape (from S. Chen, \"Machine Learning Algorithms For Beginners with Code Examples in Python\", June 2020)\n",
    "      There are many possible diagrams that one deep learning might prepare to show the machine learning landscape, including ones with a larger emphasis on text and knowledge graphs. Most all schematics of the field show a basic split between supervised learning and unsupervised learning, (sometimes with reinforcement learning as another main branch), with the main difference being that supervised approaches iterate to achieve statistical fit with pre-determined labels, whereas unsupervised is unlabeled. Accurate labeling can be costly and time consuming. Note that the idea of 'classification' is a supervised one, 'clustering' a notion of unsupervised.\n",
    "      We will include a 'standard' machine learning library in our proposed toolkit, the selection of which I discuss below. However, the most evaluation time I spent in researching these installments was directed to the idea of knowledge representation and embeddings applicable to graphs. Graphs pose a number of differences and challenges to standard machine learning. They have only been a recent (5 yr) focus in machine learning, which is also rapidly changing over time.\n",
    "      \"\"\"\n",
    "\n",
    "move_names = list(ner.move_names)\n",
    "assert nlp.get_pipe(\"ner\").move_names == move_names\n",
    "doc = nlp(text)\n",
    "for ent in doc.ents:\n",
    "    print(ent.label_, ent.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see these tags in context to the original text, we can also invoke the visual annotator available in <code>spaCy</code>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">With this installment of \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    the Cooking with Python\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia series\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " we move into \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Part VI\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " of \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    seven\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " parts, a part with the bulk of the analytical and \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " (that is, &quot;data science&quot;) discussion, and the last part where significant code is developed and documented. At the conclusion of this part, which itself has \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    11\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " installments, we have \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    four\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " installments to wrap up the series and provide a consistent \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    roadmap\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " to the entire project.</br>      Knowledge graphs are unique information artifacts, and \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " is further unique in terms of its consistent and logical construction as well as its incorporation of significant text content via Wikipedia pages. These characteristics provide unique value for \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", but it is also a combination not duplicated anywhere else in the data science ecosystem. \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    One\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of the objectives, therefore, of this part of our CWPK series is the creation of some baseline knowledge representations useful to data science aims that capture these unique characteristics.</br>      \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia's\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " (or any knowledge graph constructed in a similar manner) combination of characteristics make it a powerful resource in \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    three\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " areas of data science and \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ". \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    First\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       ", the nearly universal scope and degree of topic coverage with \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    about 56,000\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " concepts, logically organized into typologies with a high degree of disjointedness, means that accurate 'slices' or training sets may be extracted from \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " nearly instantaneously. Labeled training sets are one of the most time consuming and expensive activities in doing supervised \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ". We can extract these nearly for free from \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ". Further, with its links to \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    tens of millions\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of entities in its mapped knowledge bases such as Wikidata, literally \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    tens of thousands\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of conceptual entities in \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " can be the retrieval points to nucleate training sets for fine-grained entity recognition.</br>      \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Second\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       ", \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    80%\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERCENT</span>\n",
       "</mark>\n",
       " of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       "'s concepts are mapped to Wikipedia articles. While many Wikipedia-based \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    word embedding\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " models exist, the ones in \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " are logically categorized and have rough equivalence in terms of scope and prominence, hopefully providing cleaner topic 'signals'. To probe these assertions, we will create a unique \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       "-based \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    word embedding\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " corpus that also leverages labels for items of structural importance, such as typology membership. We will use this corpus in many of our tests and as a general focus in our training sets.</br>      And, \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    third\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       ", perhaps the most important area, knowledge graphs offer unique structures and challenges for \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", especially innovations in geometric, heterogeneous methods for \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    deep learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ". The \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    first\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       " generation of deep \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " was designed for grid-patterned data and matrices through approaches such as deep \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    neural networks\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", convolutional \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    neural networks\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " (\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    CNN\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " ), or recurrent \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    neural networks\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " (\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    RNN\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "). The 'deep' appelation comes from having multiple calculated, intermediate layers of transformations between the grid inputs and outputs for the model. \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Graphs\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ", on the other hand, are heterogeneous between nodes and edges. They may be directed (subsumptive) in nature. And, for knowledge graphs, they have much labeling and annotation, including varying degrees of attribute completeness. Language embedding, itself often a product of \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    deep learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", enables the efficient incorporation of text. It is only in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    the past five years\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " that concerted attention has been devoted to better capturing this feature richness for knowledge graphs.</br>      The \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    eleven\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " installments in this part will look in more depth at networks and graphs, focus on how to create training sets and embeddings for the learners, discuss some natural language packages and uses, and then look in depth at 'standard' machine learners and deep learners. We will install the \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    first\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       " generation of deep graph learners and then explore some on the cutting edge. We will test many use cases, but will also try to invoke classifiers across this spectrum so that we can draw some general conclusions.</br>      The material below introduces and tees up these topics. We describe leading \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Python\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " packages for data science, and how we have architected our own approach, We have picked a particular Python \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " framework, \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    PyTorch\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ", to which we will then tie \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    four\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " different \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    NLP\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    deep learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " libraries. We devote \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    two\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " installments each to these \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    four\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " libraries. The use cases we document across these installments are in addition to the existing ones we have in \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Clojure\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " posted online.</br>      So, we think we have an interesting suite of benefits to cover in this part, some arising from being based on \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    KBpedia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " and some arising from the nature of knowledge graphs. On the other hand, due to the relative immaturity of the field, we are still actively learning and innovating around the juncture of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    AI\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " and knowledge graphs. Thus, \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    one\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of the reasons we emphasize \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Python\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " 'ecosystems' and 'frameworks' in this part is to be better prepared to incorporate those innovations and learnings to come.</br>      Background</br>      One of the \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    first\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       " prototypes of \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " comes from the statistician \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Ronald Fisher\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    the 1930s\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " regarding how to classify \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Iris\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " species based on the attributes of their flowers. It was a multivariate data example using the method we \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    today\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " call \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    linear discriminant analysis\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ". This classic example is still taught. But many \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    dozens\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of new algorithms and combined approaches have joined the \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " field since then.</br>      Figure \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    1\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " below is \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    one\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " way to characterize the field, with \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ML\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " standing for \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #c887fb; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    DL\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">NORP</span>\n",
       "</mark>\n",
       " for \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    deep learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", with this one oriented to sub-fields in which some \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Python\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " package already exists:</br>      Machine Learning Landscape</br>      Figure 1: Machine Learning Landscape (from \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    S. Chen\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ", &quot;\n",
       "<mark class=\"entity\" style=\"background: #f0d0ff; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Machine Learning Algorithms For Beginners with Code Examples in Python\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">WORK_OF_ART</span>\n",
       "</mark>\n",
       "&quot;, \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    June 2020\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ")</br>      There are many possible diagrams that \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    one\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    deep learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " might prepare to show the \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " landscape, including ones with a larger emphasis on text and knowledge graphs. Most all schematics of the field show a basic split between \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    supervised learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    unsupervised learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", (sometimes with \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    reinforcement learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " as another main branch), with the main difference being that supervised approaches iterate to achieve statistical fit with pre-determined labels, whereas unsupervised is unlabeled. Accurate labeling can be costly and time consuming. Note that the idea of 'classification' is a supervised one, 'clustering' a notion of unsupervised.</br>      We will include a 'standard' \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       " library in our proposed toolkit, the selection of which I discuss below. However, the most evaluation time I spent in researching these installments was directed to the idea of knowledge representation and embeddings applicable to graphs. \n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Graphs\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       " pose a number of differences and challenges to standard \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ". They have only been a recent (\n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    5 yr\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">QUANTITY</span>\n",
       "</mark>\n",
       ") focus in \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    machine learning\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ML</span>\n",
       "</mark>\n",
       ", which is also rapidly changing over time.\n",
       "      </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "#import en_core_web_sm\n",
    "\n",
    "\n",
    "model = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\results\\en_core_ml'\n",
    "\n",
    "nlp = spacy.load(model)\n",
    "#nlp = en_core_web_sm.load()\n",
    "\n",
    "text = \"\"\"With this installment of the Cooking with Python and KBpedia series we move into Part VI of seven parts, a part with the bulk of the analytical and machine learning (that is, \"data science\") discussion, and the last part where significant code is developed and documented. At the conclusion of this part, which itself has 11 installments, we have four installments to wrap up the series and provide a consistent roadmap to the entire project.\n",
    "      Knowledge graphs are unique information artifacts, and KBpedia is further unique in terms of its consistent and logical construction as well as its incorporation of significant text content via Wikipedia pages. These characteristics provide unique value for KBpedia, but it is also a combination not duplicated anywhere else in the data science ecosystem. One of the objectives, therefore, of this part of our CWPK series is the creation of some baseline knowledge representations useful to data science aims that capture these unique characteristics.\n",
    "      KBpedia's (or any knowledge graph constructed in a similar manner) combination of characteristics make it a powerful resource in three areas of data science and machine learning. First, the nearly universal scope and degree of topic coverage with about 56,000 concepts, logically organized into typologies with a high degree of disjointedness, means that accurate 'slices' or training sets may be extracted from KBpedia nearly instantaneously. Labeled training sets are one of the most time consuming and expensive activities in doing supervised machine learning. We can extract these nearly for free from KBpedia. Further, with its links to tens of millions of entities in its mapped knowledge bases such as Wikidata, literally tens of thousands of conceptual entities in KBpedia can be the retrieval points to nucleate training sets for fine-grained entity recognition.\n",
    "      Second, 80% of KBpedia's concepts are mapped to Wikipedia articles. While many Wikipedia-based word embedding models exist, the ones in KBpedia are logically categorized and have rough equivalence in terms of scope and prominence, hopefully providing cleaner topic 'signals'. To probe these assertions, we will create a unique KBpedia-based word embedding corpus that also leverages labels for items of structural importance, such as typology membership. We will use this corpus in many of our tests and as a general focus in our training sets.\n",
    "      And, third, perhaps the most important area, knowledge graphs offer unique structures and challenges for machine learning, especially innovations in geometric, heterogeneous methods for deep learning. The first generation of deep machine learning was designed for grid-patterned data and matrices through approaches such as deep neural networks, convolutional neural networks (CNN ), or recurrent neural networks (RNN). The 'deep' appelation comes from having multiple calculated, intermediate layers of transformations between the grid inputs and outputs for the model. Graphs, on the other hand, are heterogeneous between nodes and edges. They may be directed (subsumptive) in nature. And, for knowledge graphs, they have much labeling and annotation, including varying degrees of attribute completeness. Language embedding, itself often a product of deep learning, enables the efficient incorporation of text. It is only in the past five years that concerted attention has been devoted to better capturing this feature richness for knowledge graphs.\n",
    "      The eleven installments in this part will look in more depth at networks and graphs, focus on how to create training sets and embeddings for the learners, discuss some natural language packages and uses, and then look in depth at 'standard' machine learners and deep learners. We will install the first generation of deep graph learners and then explore some on the cutting edge. We will test many use cases, but will also try to invoke classifiers across this spectrum so that we can draw some general conclusions.\n",
    "      The material below introduces and tees up these topics. We describe leading Python packages for data science, and how we have architected our own approach, We have picked a particular Python machine learning framework, PyTorch, to which we will then tie four different NLP and deep learning libraries. We devote two installments each to these four libraries. The use cases we document across these installments are in addition to the existing ones we have in Clojure posted online.\n",
    "      So, we think we have an interesting suite of benefits to cover in this part, some arising from being based on KBpedia and some arising from the nature of knowledge graphs. On the other hand, due to the relative immaturity of the field, we are still actively learning and innovating around the juncture of AI and knowledge graphs. Thus, one of the reasons we emphasize Python 'ecosystems' and 'frameworks' in this part is to be better prepared to incorporate those innovations and learnings to come.\n",
    "      Background\n",
    "      One of the first prototypes of machine learning comes from the statistician Ronald Fisher in the 1930s regarding how to classify Iris species based on the attributes of their flowers. It was a multivariate data example using the method we today call linear discriminant analysis. This classic example is still taught. But many dozens of new algorithms and combined approaches have joined the machine learning field since then.\n",
    "      Figure 1 below is one way to characterize the field, with ML standing for machine learning and DL for deep learning, with this one oriented to sub-fields in which some Python package already exists:\n",
    "      Machine Learning Landscape\n",
    "      Figure 1: Machine Learning Landscape (from S. Chen, \"Machine Learning Algorithms For Beginners with Code Examples in Python\", June 2020)\n",
    "      There are many possible diagrams that one deep learning might prepare to show the machine learning landscape, including ones with a larger emphasis on text and knowledge graphs. Most all schematics of the field show a basic split between supervised learning and unsupervised learning, (sometimes with reinforcement learning as another main branch), with the main difference being that supervised approaches iterate to achieve statistical fit with pre-determined labels, whereas unsupervised is unlabeled. Accurate labeling can be costly and time consuming. Note that the idea of 'classification' is a supervised one, 'clustering' a notion of unsupervised.\n",
    "      We will include a 'standard' machine learning library in our proposed toolkit, the selection of which I discuss below. However, the most evaluation time I spent in researching these installments was directed to the idea of knowledge representation and embeddings applicable to graphs. Graphs pose a number of differences and challenges to standard machine learning. They have only been a recent (5 yr) focus in machine learning, which is also rapidly changing over time.\n",
    "      \"\"\"\n",
    "\n",
    "doc = nlp(text)\n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see that our 'ML' tag has been added to the roster and other standard tags are shown.\n",
    "\n",
    "Were this to be a production version, I would spend more time updating the training examples to remove some of the misassignments and would likely add some additional ML tags specific to our work with KBpedia (as opposed to the ones strictly from Wikipedia). Nonetheless, it seems like the rule-based approach is the better one for a topic area like 'machine learning' when we have a rather complete enumeration of important instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other spaCy Functions\n",
    "There is a wealth of additional functions that might be applied to KBpedia and its uses with the <code>spaCy</code> package. For example, this simple routine shows the variety of tags and characterizations that might be retrieved from text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 0, 'paragraphs': [{'raw': 'Machine learning is fun in Iowa.', 'sentences': [{'tokens': [{'id': 0, 'orth': 'Machine', 'tag': 'NN', 'head': 1, 'dep': 'compound', 'ner': 'O'}, {'id': 1, 'orth': 'learning', 'tag': 'NN', 'head': 1, 'dep': 'nsubj', 'ner': 'O'}, {'id': 2, 'orth': 'is', 'tag': 'VBZ', 'head': 0, 'dep': 'ROOT', 'ner': 'O'}, {'id': 3, 'orth': 'fun', 'tag': 'JJ', 'head': -1, 'dep': 'attr', 'ner': 'O'}, {'id': 4, 'orth': 'in', 'tag': 'IN', 'head': -2, 'dep': 'prep', 'ner': 'O'}, {'id': 5, 'orth': 'Iowa', 'tag': 'NNP', 'head': -1, 'dep': 'pobj', 'ner': 'U-GPE'}, {'id': 6, 'orth': '.', 'tag': '.', 'head': -4, 'dep': 'punct', 'ner': 'O'}], 'brackets': []}], 'cats': []}]}\n"
     ]
    }
   ],
   "source": [
    "from spacy.gold import docs_to_json\n",
    "\n",
    "doc = nlp(\"Machine learning is fun in Iowa.\")\n",
    "json_data = docs_to_json([doc])\n",
    "print(json_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The combination of its rich functionality and pipeline abilities ensures <code>spaCy</code> is an NLP package of great capability. We could devote more write-ups to applications like topic modeling, word sense disambiguation, or relation extraction, but we need to move on in the next installment to classic machine learning. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "Here is additional documentation in support of this installment.\n",
    "\n",
    "##### Embeddings and Transformers\n",
    "- [What Are Word Embeddings for Text?](https://machinelearningmastery.com/what-are-word-embeddings/)\n",
    "- [word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Data](https://arxiv.org/pdf/2003.12590.pdf) \"Rather than focussing on distances, knowledge graph embeddings focus on establishing a correspondence between the relations of the knowledge graph and geometric relation-ships in the latent space\"\n",
    "- [Word Embeddings in Python with Spacy and Gensim](https://info.cambridgespark.com/latest/word-embeddings-in-python) helpful with many code examples\n",
    "- [Tutorial: Build your own Skip-gram Embeddings and use them in a Neural Network](https://blog.cambridgespark.com/tutorial-build-your-own-embedding-and-use-it-in-a-neural-network-e9cde4a81296) uses GenSim, skip-gram, and word2vec, but is based on NLTK and Kears, not our environment. Still, very informative\n",
    "- [Transformers from Scratch](http://peterbloem.nl/blog/transformers) is an excellent discussion of transformers with much accompanying PyTorch code\n",
    "- [spaCy meets Transformers: Fine-tune BERT, XLNet and GPT-2](https://explosion.ai/blog/spacy-transformers)\n",
    "- [What is Word Embedding | Word2Vec | GloVe](https://www.mygreatlearning.com/blog/word-embedding/).\n",
    "\n",
    "##### Text Summarization\n",
    "- [Text Generation from Knowledge Graphs with Graph Transformers](https://arxiv.org/abs/1904.02342) and its accompanying [GraphWriter](https://github.com/rikdz/GraphWriter) code \n",
    "- [Knowledge Graph Empowered Entity Description Generation](http://arxiv.org/abs/2004.14813) - structure to description generator; does not seem that easily repeatable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}