cwpk-30-extracting-annotations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#30: Extracting Annotations\n",
    "=======================================\n",
    "\n",
    "Everything Can Be Annotated in a Knowledge Graph\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "We've seen in the previous two installments of this [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series various ways to specify a subset population for driving an iterative process for extracting structure from [KBpedia](https://kbpedia.org/). We're going to retain that iterative approach, only change it now to extract annotations. Classes, properties, and instances (individuals) may all be annotated in [OWL](https://en.wikipedia.org/wiki/Web_Ontology_Language). We thus need to derive generalized approaches that can apply to any entity in a knowledge graph.\n",
    "\n",
    "Annotations are information applied to a given entity in order to point to it, describe it, or identify it. As a best practices matter, there are certain fields we recommend be universally applied to annotate any given entity:\n",
    "\n",
    "- A preferred label (<code>prefLabel</code>) that is the standard name or title for a thing\n",
    "- A multiple of alternative labels (<code>altLabel</code>) that capture any of the ways a given thing may be referred to, including synonyms, acronyms, jargon, etc. \n",
    "- A definition of the thing (<code>definition</code>)\n",
    "- All labels should be tagged with a language tag in order to more readily support translation and use in multiple languages.\n",
    "\n",
    "We may also find comments or notes associated with particular items. Further, in the case of object or data properties, we may have additional characterizations such as <code>domain</code> or <code>range</code> or functionality assigned to the item. We could have retrieved these characterizations as part of our structural extractions, but decided to include them rather in an annotation extraction pass (even though those characterizations are not annotative)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Items to be Extracted During Annotation Pass\n",
    "We can thus assemble up a list of items that may be extracted during an annotation extraction pass. We could do these extractions in parts, since that is often the better approach during the inverse process of building our knowledge graph. However, given the number of annotation and related items that may be extracted, and the number of combinations of same, we decide as a matter of simplicity to extract all such information as a single record for each subject entity. We can later manipulate the large flat files so generated if we need to focus on subsets of them. We may revisit this question once we tackle the build side of this [roundtripping](https://en.wikipedia.org/wiki/Round-trip_format_conversion) process.\n",
    "\n",
    "Some of the items that we will extract have multiple entries per subject. Parental class is one such item, as are alternative labels, which may number into the tens for a rather complete characterization. From our experience in the last installments we know we will need to set up some inner loops to accommodate such multiple entries. So, with these understandings, we can now compile up a list of items that may be extracted on an annotation extraction pass, including whether the item is limited to a single entry, or may have many:\n",
    "\n",
    "- IRI fragment name: single\n",
    "- <code>prefLabel</code>: single\n",
    "- <code>altLabel</code>: many\n",
    "- superclass: many\n",
    "- <code>definition</code>: single\n",
    "- <code>editorialNote</code>: many\n",
    "- mapping properties: many (a characterization that will grow over time)\n",
    "- <code>comment</code>: many\n",
    "- <code>domain</code>: single (object and data properties, only)\n",
    "- <code>range</code>: single (object and data properties, only)\n",
    "- functional type: single (object and data properties, only)\n",
    "\n",
    "So, we decide to develop two variants of our code block. A standard one, and an expanded one that includes the object and data property additions. The IRI fragment name is the alias used internally in our Python programs and what gets concatenated with the base IRI to form the full IRI for the entity. \n",
    "\n",
    "Also, to maintain the idea of a single line per subject entity, we decide that: 1) we will separate multiple entries for a given item with the '||' (\"double pipe\") separator, which we use because it is never used in the wild and it is easy to spot when scanning code; and 2) we will not use full IRIs in order to aid record readability.\n",
    "\n",
    "(BTW, if we decide over time to add other standard characterizations to our items we will adjust our routines accordingly.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting and Load\n",
    "We again begin with our standard opening routine, except we have now substituted 'kbpedia' for 'main' in the first line, to make our reference going forward more specific:\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service (the 'raw' version) or local files. The example below is based on using local files (though replace with your own local directory specification). If loading from MyBinder, replace with the lines that are commented (<code>#</code>) out.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "kbpedia = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'\n",
    "# kbpedia = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl'\n",
    "skos_file = 'http://www.w3.org/2004/02/skos/core' \n",
    "kko_file = 'C:/1-PythonProjects/kbpedia/sandbox/kko.owl'\n",
    "# kko_file = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl'\n",
    "\n",
    "\n",
    "from owlready2 import *\n",
    "world = World()\n",
    "kb = world.get_ontology(kbpedia).load()\n",
    "rc = kb.get_namespace('http://kbpedia.org/kko/rc/')               \n",
    "\n",
    "skos = world.get_ontology(skos_file).load()\n",
    "kb.imported_ontologies.append(skos)\n",
    "core = world.get_namespace('http://www.w3.org/2004/02/skos/core#')\n",
    "\n",
    "kko = world.get_ontology(kko_file).load()\n",
    "kb.imported_ontologies.append(kko)\n",
    "kko = kb.get_namespace('http://kbpedia.org/ontologies/kko#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like always, we execute each cell as we progress down this notebook page by pressing <code>shift+enter</code> for the highlighted cell or by choosing Run from the notebook menu.\n",
    "\n",
    "### Basic Extraction Set-up\n",
    "We tackle the smaller (non-property) variant of our code block first, treating the extracted items listed above as the members of a Python set specification. We also choose to prefix our variables with <code>annot_</code>. We will first start with a single item, foregoing the loop for the moment, to test if we have gotten our correspondences right. For the class set-up we'll use the relatively small <code>rc.Luggage</code> class. (You may substitute any KBpedia RC as this item.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['baggage'], '', ['bag', 'bags', 'luggage'], ['This ‘class’ product category is drawn from UNSPSC, readily converted into other widely used systems. This product ‘class’ is an aggregate of luggage. This product category corresponds to the UNSPSC code: 53121500.'], []]\n"
     ]
    }
   ],
   "source": [
    "s_item = rc.Luggage\n",
    "annot_pref = s_item.prefLabel\n",
    "annot_sup  = ''\n",
    "# annot_sup  = s_item.superclass  # maybe it should be is_a\n",
    "annot_alt  = s_item.altLabel\n",
    "annot_def  = s_item.definition\n",
    "annot_note = s_item.editorialNote\n",
    "annot = [annot_pref, annot_sup, annot_alt, annot_def, annot_note]\n",
    "print(annot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to add a few items to deal with specific property characteristics including <code>domain</code>, <code>range</code>, and functional type (which is blank in all of our cases):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['representations'], [owl.AnnotationProperty], [], [], '', ['annotations', 'indexicals', 'metadata'], ['Pointers or indicators, including symbolic ones such as text or URLs, that draw attention to the actual or dynamic object.'], []]\n"
     ]
    }
   ],
   "source": [
    "s_item = kko.representations\n",
    "annot_pref = s_item.prefLabel\n",
    "annot_sup  = s_item.is_a\n",
    "annot_dom  = s_item.domain\n",
    "annot_rng  = s_item.range\n",
    "annot_func = ''\n",
    "annot_alt  = s_item.altLabel\n",
    "annot_def  = s_item.definition\n",
    "annot_note = s_item.editorialNote\n",
    "annot = [annot_pref, annot_sup, annot_dom, annot_rng, annot_func, annot_alt, annot_def, annot_note]\n",
    "print(annot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KBpedia does not use functional properties at present. I leave a placeholder above, but have not worked out the [owlready2](https://owlready2.readthedocs.io/en/latest/intro.html) access methods.\n",
    "\n",
    "### Working Out the Code Block\n",
    "A quick inspection of these outputs flags a few areas of concern. We see that items are often enclosed in square brackets (a set notation in Python), we have many quoted items, and we have (as we knew) mutiple entries for some fields, especially <code>altLabel</code> and parents. In order to test our code block out, we will need to have a test set loaded. We decide to keep on with <code>rc.Luggage</code>, but I throw in a length count. You can substitute any non-leaf RC into the code if you want a larger or smaller or different domain test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root = rc.Luggage\n",
    "s_set=root.descendants()\n",
    "\n",
    "len(s_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the iteration part for the multiple entries, we begin with the code blocks used for the inner loops dealing with the structural backbone issues in [**CWPK #28**](https://www.mkbergman.com/2363/cwpk-28-extracting-structure-for-typologies/) and [**CWPK #29**](https://www.mkbergman.com/2364/cwpk-29-extracting-object-and-data-properties/). But the purpose of tracing inheritance is different than retrieving values for multiple attributes of a single entity. Maybe we should tackle what seems to be an easier concern to remove the enclosing brackets ('[ ]').\n",
    "\n",
    "I also decide as we test out these code blocks that I would shorten the variable names to reduce the amount of typing and to reflect a more general procedure. So, all of the <code>annot_</code> prefixes from above become <code>a_</code>.\n",
    "\n",
    "Poking around I first find a string replacement example, followed by the <code>.join</code> method for strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for s_item in s_set:\n",
    "  a_pref = s_item.prefLabel\n",
    "  a_sup  = s_item.is_a\n",
    "  a_alt  = s_item.altLabel\n",
    "  a_def  = s_item.definition\n",
    "  a_note = s_item.editorialNote\n",
    "  a_     = [a_pref,a_sup,a_alt,a_def,a_note]\n",
    "  def listToStringWithoutBrackets(a_):\n",
    "    return str(a_).replace('[','').replace(']','')\n",
    "  listToStringWithoutBrackets(a_)\n",
    "  print( ','.join( repr(e) for e in a_ ) )\n",
    "len(s_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I try another string substitution example with similarly disappointing results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for s_item in s_set:\n",
    "  a_pref = s_item.prefLabel\n",
    "  a_sup  = s_item.is_a\n",
    "  a_alt  = s_item.altLabel\n",
    "  a_def  = s_item.definition\n",
    "  a_note = s_item.editorialNote\n",
    "  a_     = [s_item,a_pref,a_sup,a_alt,a_def,a_note]\n",
    "  a_     = [str(i) for i in a_]\n",
    "  a_out  = ','.join(a_).strip('[]') \n",
    "  print(a_out)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason these, and multiple other string approaches failed, was that we are dealing with results sets with multiple entries. It seemed like the safest way to ensure the fields were treated as strings was to explicitly declare them as such, and then manipulate the string directly. So, in the code below, we grab the property from the entity, convert it to a string, and then remove the first and last characters of the entire string, which in our case are the brackets. Note in this test code that I also (temporarily) comment out the two fields where we have possibly multiple items that we want to loop over and concatenate into a single string entry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_item = ''\n",
    "for s_item in s_set:\n",
    "  a_pref = s_item.prefLabel\n",
    "  a_pref  = str(a_pref)[1:-1]                   # this is one way to remove opening and closing characters ([ ])\n",
    "#  a_sup  = s_item.is_a\n",
    "#  a_alt  = s_item.altLabel\n",
    "  a_def  = s_item.definition\n",
    "  a_def  = str(a_def)[1:-1]\n",
    "  a_note = s_item.editorialNote\n",
    "  a_note  = str(a_note)[1:-1]\n",
    "  print(s_item,a_pref,a_sup,a_alt,a_def,a_note, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We still see brackets in the listing, but those are for the two properties we commented out. All other brackets are now gone. While I really do not like repeating the same bracket removal code multiple times, it works, and after spending perhaps more time than I care to admit trying to find a more elegant solution, I decide to accept the workable over the perfect. I am hoping when we loop over the elements for the two fields commented out that we will be extracting the element from each pass of the loop, and thus via processiing will see the removal of their brackets. (That apparently is what happens in the loop steps below.)\n",
    "\n",
    "Now, it is time to tackle the harder question of collapsing (also called 'flattening') a field with multiple entries. The basic idea of this inner loop is to treat all elements as strings, loop over the multiple elements, grab the first one and end if there is only one, but to continue looping if there is more than one until the number of elements is met, and to add a 'double pipe' ('||') character string to the previously built elements before concatenating the current element. This order of putting the delimiter at the beginning of each loop result is to make sure our final string with all concatenated results does not end with a delimiter. The skipping of the first pass means no delimiter is added at the beginning of the first element, also good if there is only one element for a given entity, which is often the case.\n",
    "\n",
    "There are very robust <code>for</code> and <code>while</code> operators in Python. The one I settled on for this example uses an <code>id,enumerate</code> tuple where we get both the current element item and its numeric index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_item = ''\n",
    "for s_item in s_set:\n",
    "  a_pref = s_item.prefLabel\n",
    "  a_pref  = str(a_pref)[1:-1]\n",
    "#  a_sup  = s_item.is_a\n",
    "  a_alt  = s_item.altLabel\n",
    "  for a_id, a in enumerate(a_alt):            # here is the added inner loop as explained in text\n",
    "    a_item = str(a)\n",
    "    if a_id > 1:\n",
    "        a_item = a_item + '||' + str(a)\n",
    "  a_alt  = a_item\n",
    "  a_def  = s_item.definition\n",
    "  a_def  = str(a_def)[1:-1]\n",
    "  a_note = s_item.editorialNote\n",
    "  a_note  = str(a_note)[1:-1]\n",
    "  print(s_item,a_pref,a_sup,a_alt,a_def,a_note, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the inner loop example is working we can duplicate the approach for the other inner loop and move on to putting a full working code block together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class Annotations\n",
    "OK, so we appear ready to start finalizing the code block. We will start with class annotations because they have fewer fields to capture. The first step we want to do is to remove the pesky <code>rc.</code> namespace prefix in our output. Remember, this came from a tip in our last installment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_using_label(entity):\n",
    "    return entity.label.first() or entity.name\n",
    "\n",
    "set_render_func(render_using_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(How to set it back to the default is described in the prior installment.)\n",
    "\n",
    "We also pick a class and its descendants to use in our prototype example. I also add a <code>len</code> statement in the code to indicate how many classes we will be processing in this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "root = rc.Luggage\n",
    "s_set=root.descendants()\n",
    "\n",
    "len(s_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now expand our code block to set our initial iterator to an empty string, fix (remove) the brackets, and process the two inner loops of the <code>altLabels</code> and parent classes putting the \"double pipe\" ('||') between entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EveningBag,'evening bag',Purse,evening bags,'The collection of all evening bags. A type of Purse. The collection EveningBag is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "Gucci,'Gucci',Luggage,GUCCI,'Gucci (/ɡuːtʃi/; Italian pronunciation: [ˈɡuttʃi]) is an Italian luxury brand of fashion and leather goods, part of the Gucci Group, which is owned by the French holding company Kering. Gucci was founded by Guccio Gucci in Florence in 1921.Gucci generated about €4.2 billion in revenue worldwide in 2008 according to BusinessWeek and climbed to 41st position in the magazine\\'s annual 2009 \\\\\"Top Global 100 Brands\\\\\" chart created by Interbrand; it ranked retained that rank in Interbrand\\'s 2014 index. Gucci is also the biggest-selling Italian brand. Gucci operates about 278 directly operated stores worldwide as of September 2009, and it wholesales its products through franchisees and upscale department stores. In the year 2013, the brand was valued at US$12.1 billion, with sales of US$4.7 billion.',\n",
      "Briefcase,'briefcase',CarryingCase||Device-OfficeProduct-NonConsumable||OfficeProductMarketCategory||Box-Container,Attache cases||Attaché case||Attaché cases||Brief case||Handlebox||Portfolio (briefcase)||briefcases,'The collection of all briefcases, which are small portable cases designed to carry hardcopies of documents. A type of Luggage and Box_Container. The collection Briefcase is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType. This ‘commodity’ product category is drawn from UNSPSC, readily converted into other widely used systems. This commodity category is for briefcase. This product category corresponds to the UNSPSC code: 53121701.',\n",
      "DuffelBag,'duffel bag',Luggage,Duffel bags||Duffle||Duffle Bag||Duffle bag||Dufflebag||Kit bag||Kit-bag||Seabag||The dufflebag||duffel bags,'The collection of all duffel bags. A type of Luggage. The collection DuffelBag is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "ShoulderBag,'shoulder bag',Luggage,shoulder bags,'The collection of all shoulder bags -- luggage bags with straps for carrying over the shoulder. A type of Bag and Luggage. The concept ShoulderBag is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "GolfBag,'golf bag',CarryingCase,golf bags,'Cylindrical bag about one meter long, open on one end; for carrying golf clubs. Has shoulder strap.',\n",
      "SkiCarrier,'ski carrier',MechanicalDevice||ContainerArtifact,ski carriers,'The collection of all ski carriers. A type of ContainerArtifact and MechanicalDevice. The concept SkiCarrier is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "Backpack,'knapsack',PurseHandbagBag||SomethingToWear||Luggage,Back packs||Backbacks||Backpacks||Book bags||Book sack||Bookbags||Booksack||Day pack||Daypack||Haver sack||Haver sacks||Haversacks||Knap sack||Knap sacks||Knapsack||Pack sack||Pack sacks||Packsack||Packsacks||Ruck sack||Rucksack||Rucksacks||Schoolbag||backpack||backpacks||knapsacks||rucksack||rucksacks,'The collection of all backpacks. A type of Luggage, NonPoweredDevice, and SomethingToWear. The collection Backpack is an ArtifactTypeByFunction and a SpatiallyDisjointObjectType.',\n",
      "Luggage,'baggage',TransportationContainerProduct,bags||luggage,'This ‘class’ product category is drawn from UNSPSC, readily converted into other widely used systems. This product ‘class’ is an aggregate of luggage. This product category corresponds to the UNSPSC code: 53121500.',\n",
      "TravelAccessory,'travel accessory',Luggage,travel gear,'Instances are tangible products which are intended for use during travelling. Often they are lighter-weight, more-compact versions of products people normally use at home.',\n",
      "DiaperBag,'diaper bag',ShoulderBag,Nappy bag||diaper bags,'The collection of all diaper bags. A DiaperBag is a bag which the caretaker of a baby may put Diapers, wipes, and other baby-related products into to bring with him or her when traveling with a HumanInfant or HumanToddler.',\n",
      "LuggageTag,'luggage tag',TravelAccessory,Luggage tags||luggage tags,'The collection of all luggage tags. A type of Tag_IBO and TravelAccessory. The collection LuggageTag is an #$AerodromeCollection, a SpatiallyDisjointObjectType, and an ArtifactTypeByGenericCategory.',\n",
      "WomensToteBag,'tote',PurseHandbagBag,satchels||totes||women's satchel||women's satchels||women's tote||women's tote bag||women's tote bags||women's totes,'The collection of all totes. A type of WomensBag. The collection WomensToteBag is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "LuggageSet,'luggage ensemble',Luggage,luggage set||luggage sets,'The collection of all luggage ensembles. A type of group of baggage, Artifact, and DurableGood. LuggageSet is an ArtifactTypeByGenericCategory.',\n",
      "Suitcase,'suitcase',Luggage,Suitcases||Swedish lunchbox||Travel bag||Trolley case||Valise||grip||grips||suitcases||travelling bag||travelling bags||valise||valises,'Retangular piece of luggage with handle that is used when you are on a trip.',\n",
      "PurseHandbagBag,'purse or handbag or bag',ContainerArtifact||LuggageHandbagPack,Clutch (handbag)||Evening bag||Hand bag||Hand-bag||Hand-bags||Handbags||Man bag||Man purse||Man-bag||Manbag||Manpurse,'This ‘class’ product category is drawn from UNSPSC, readily converted into other widely used systems. This product ‘class’ is an aggregate of purse or handbag or bag. This product category corresponds to the UNSPSC code: 53121600.',\n",
      "PortfolioCase,'portfolio case',CarryingCase,portfolio cases||portfolios,'The collection of all portfolio cases, which are cases desinged for carrying portfolios of art or writing. A type of CarryingCase. The collection PortfolioCase is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "Purse,'purse',NonPoweredDevice||PersonalDevice,Purses||purses,\"The collection of all purses. A type of women's clothing accessory, PersonalDevice, and WomensBag. The collection Purse is a SomethingToWearTypeByGenericCategory and a SpatiallyDisjointObjectType.\",\n",
      "ComputerBag,'computer carrying case',Luggage,computer bags||computer carrying cases||laptop bag||laptop bags||laptop carrying case||laptop carrying cases||laptop case||laptop cases,'The collection of all computer carrying cases. A type of accessories for laptop computers and CarryingCase. The collection ComputerBag is an ArtifactTypeByFunction and a SpatiallyDisjointObjectType.',\n",
      "CarryOnLuggage,'carry on luggage',Luggage,carry on bag||carry on bags||carry-on||carry-on bag||carry-on baggage||carry-on bags||carry-on luggage,'The collection of all carry-on luggage. A type of Luggage. The collection CarryOnLuggage is an ArtifactTypeByFunction, a SpatiallyDisjointObjectType, and an #$AerodromeCollection.',\n",
      "VoltageConverter,'voltage converter',TravelAccessory,voltage adapters||voltage adaptor||voltage adaptors||voltage converters,'The collection of all voltage converters. A type of TravelAccessory, ElectricalDevice, and travel accessory. VoltageConverter is a SpatiallyDisjointObjectType.',\n",
      "CarryingCase,'carrying case',Luggage,carrying cases,'The collection of all carrying cases. A type of Luggage and Box_Container. The collection CarryingCase is an ArtifactTypeByGenericCategory and a SpatiallyDisjointObjectType.',\n",
      "GarmentBag,'hanging bag',Luggage,garment bags||hanging bags||suit bag,'The collection of all garment bags. A type of Luggage. The collection GarmentBag is an ArtifactTypeByFunction and a SpatiallyDisjointObjectType.',\n",
      "CoinPurse,'coin purse',PurseHandbagBag,[],'This ‘commodity’ product category is drawn from UNSPSC, readily converted into other widely used systems. This commodity category is for coin purse. This product category corresponds to the UNSPSC code: 53121605.',\n",
      "LuggageOrganizer,'luggage organizer',Organizer||ContainerArtifact,luggage organisers||luggage organizers,'This is the collection of products which fit into luggage whose purpose is to structure the space inside in such as way that the items stored therein are more organized and perhaps better protected.',\n"
     ]
    }
   ],
   "source": [
    "a_item = ''\n",
    "for s_item in s_set:\n",
    "  a_pref = s_item.prefLabel\n",
    "  a_pref  = str(a_pref)[1:-1]\n",
    "  a_sup  = s_item.is_a\n",
    "  for a_id, a in enumerate(a_sup): \n",
    "    a_item = str(a)\n",
    "    if a_id > 1:\n",
    "        a_item = a_sup + '||' + str(a)\n",
    "    a_sup  = a_item\n",
    "  a_alt  = s_item.altLabel\n",
    "  for a_id, a in enumerate(a_alt): \n",
    "    a_item = str(a)\n",
    "    if a_id > 1:\n",
    "        a_item = a_alt + '||' + str(a)\n",
    "    a_alt  = a_item\n",
    "  a_def  = s_item.definition\n",
    "  a_def  = str(a_def)[1:-1]\n",
    "  a_note = s_item.editorialNote\n",
    "  a_note  = str(a_note)[1:-1]\n",
    "  print(s_item,a_pref,a_sup,a_alt,a_def,a_note, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The routine now seems to be working how we want it, so we move on to accommodate the properties as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Property Annotations\n",
    "Again, we set the renderer to the 'clean' setting and now pick a property and its sub-properties to populate our working set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_using_label(entity):\n",
    "    return entity.label.first() or entity.name\n",
    "\n",
    "set_render_func(render_using_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2901"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root = kko.representations\n",
    "p_set=root.descendants()\n",
    "\n",
    "len(p_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example has nearly 3000 sub-properties! That should make for an interesting example. We add our three new properties to the prior code block. We also make another change, which is to substitute the <code>p_</code> prefix (for properties) over the prior <code>s_</code> prefix for subject (classes or individual):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_item = ''\n",
    "for p_item in p_set:\n",
    "  a_pref = p_item.prefLabel\n",
    "  a_pref  = str(a_pref)[1:-1]\n",
    "  a_sup  = p_item.is_a\n",
    "  for a_id, a in enumerate(a_sup): \n",
    "    a_item = str(a)\n",
    "    if a_id > 1:\n",
    "        a_item = a_item + '||' + str(a)\n",
    "  a_sup  = a_item\n",
    "  a_dom  = p_item.domain\n",
    "  a_dom  = str(a_dom)[1:-1]\n",
    "  a_rng  = p_item.range\n",
    "  a_rng  = str(a_rng)[1:-1]\n",
    "  a_func = ''\n",
    "  a_alt  = p_item.altLabel\n",
    "  for a_id, a in enumerate(a_alt): \n",
    "    a_item = str(a)\n",
    "    if a_id > 1:\n",
    "        a_item = a_item + '||' + str(a)\n",
    "  a_alt  = a_item\n",
    "  a_def  = p_item.definition\n",
    "  a_def  = str(a_def)[1:-1]\n",
    "  a_note = p_item.editorialNote\n",
    "  a_note  = str(a_note)[1:-1]\n",
    "  print(p_item,a_pref,a_sup,a_dom,a_rng,a_func,a_alt,a_def,a_note, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! It seems that our basic annotation retrieval mechanisms are working properly. \n",
    "\n",
    "You may have noted the <code>sep=','</code> argument in the <code>print</code> statement. It means to add a comma separator between the output variables in the listing, a useful addition in Python 3 especially given our reliance on comma-separated value (CSV) files.\n",
    "\n",
    "We are now largely done with the logic of our extractors. But, before we get to how to assemble the pieces in a working module, it is time for us to take a brief detour to learn about naming and writing output and saving to and reading from files. Since we will be using CSV files heavily, we also work that into next installment's discussion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "The routines in this installment required much background reading and examples having to do with Python loops and string processing. Here are a few I found informative for today's **CWPK** installment:\n",
    "\n",
    "- Nice [DataCamp discussion](https://www.datacamp.com/community/tutorials/18-most-common-python-list-questions-learn-python) of lists\n",
    "- [Building CSV strings in Python](https://levelup.gitconnected.com/building-csv-strings-in-python-32934aed5a9e)\n",
    "- [Strings, lists, tuples](http://www.openbookproject.net/books/bpp4awd/ch03.html)\n",
    "- [Python lists and list manipulation](https://towardsdatascience.com/python-basics-6-lists-and-list-manipulation-a56be62b1f95).\n",
    "\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}