Examples/Filtering Attributes for API Queries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# osdatahub Examples - Filtering\n",
    "\n",
    "This notebook contains examples of how to add filtering to your OS DataHub queries.\n",
    "\n",
    "In addition to filtering Product, and Extent, it is possible to query the API to only\n",
    "return features with certain attributes. These filtering methods will make it easier to access\n",
    "only the features that you care about.\n",
    "\n",
    "## Features API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Each product contains  different properties that provide important additional detail to the dataset.\n",
    "The free OpenData products usually contain a lot fewer properties than the paid-for Premium datasets.\n",
    "\n",
    "If you would like to find out the different properties that are assigned to each Feature, you can use the\n",
    "`DescribeFeatureType` API endpoint. This endpoint is not currently implemented by the osdatahub Python library,\n",
    "but you can find out more about calling the API directly [here](https://osdatahub.os.uk/docs/wfs/technicalSpecification).\n",
    "\n",
    "Here, we will use two examples - one for the free feature \"Zoomstack Local Buildings\", and one for\n",
    "the premium feature \"Topographic Area\".\n",
    "\n",
    "For Features API, there are a variety of filters to pick from that are [listed in the docs](https://osdatahub.readthedocs.io/en/latest/Utilities.html#module-osdatahub.filters).\n",
    "\n",
    "These are all OGC Standard XML filter parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a couple ways of combining these filters. One is passing multiple to the `add_filters` method, or calling it multiple times. These will be joined using an \"and\" operator, meaning all given queries must be met for a feature to be returned.\n",
    "\n",
    "The other way to combine filters is to use either the `filter_and`/`filter_or` methods or bitwise 'and' and 'or' operators, as shown below:\n",
    "\n",
    "```\n",
    "filter1, filter2, filter3 = ...\n",
    "\n",
    "# select features matching any\n",
    "filter = filter_or(filter1, filter2, filter3)\n",
    "filter = filter1 | filter2 | filter3\n",
    "filter = filter1 | filter_or(filter2, filter3)\n",
    "\n",
    "# select features matching all\n",
    "filter = filter_and(filter1, filter2, filter3)\n",
    "filter = filter1 & filter2 & filter3\n",
    "filter = filter_and(filter1, filter2 & filter3)\n",
    "\n",
    "# or some other combination of operators\n",
    "filter = (filter1 | filter2) & filter3\n",
    "filter = filter_or(filter1 & filter2, filter3)\n",
    "```\n",
    "\n",
    "The resulting filter can be passed straight to `add_filters`, like any other filter:\n",
    "\n",
    "```\n",
    "features.add_filters(filter1 | filter2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Example #1 - Zoomstack Local Buildings\n",
    "\n",
    "The first step in filtering your API query is understanding which properties are available. To do this, we use the\n",
    "`DescribeFeatureType` API endpoint described in the previous section. We ran the following query to find all available\n",
    "properties: (remember to add your own API key to the query)\n",
    "\n",
    "`https://api.os.uk/features/v1/wfs?service=wfs&version=2.0.0&request=DescribeFeatureTypetypeNames=Zoomstack_localbuildings&key=your-api-key-here`\n",
    "\n",
    "The query returns the following available properties:\n",
    "\n",
    "* OBJECTID\n",
    "* UUID\n",
    "* SHAPE\n",
    "* SHAPE_Length\n",
    "* SHAPE_Area\n",
    "\n",
    "Now that we know what we can filter by, we can begin to construct our query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Setup\n",
    "\n",
    "from osdatahub import Extent\n",
    "from osdatahub.FeaturesAPI import FeaturesAPI\n",
    "from osdatahub.filters import *\n",
    "from os import environ\n",
    "\n",
    "key = environ.get(\"OS_API_KEY\") # you can also replace this line with the API key itself\n",
    "crs = \"EPSG:27700\"\n",
    "product = \"zoomstack_local_buildings\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Define an extent around Hammersmith, London\n",
    "W, S = (521202 , 177370)\n",
    "E, N = (523546 , 179409)\n",
    "extent = Extent.from_bbox((W, S, E, N), crs=crs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now that we have a key, a product, and an extent, we can create a FeaturesAPI object.\n",
    "\n",
    "As a baseline, we will also run the query and see the number of returned features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number features returned:  350\n"
     ]
    }
   ],
   "source": [
    "features = FeaturesAPI(key, product, extent)\n",
    "results = features.query(limit=350)\n",
    "print(\"Number features returned: \", len(results[\"features\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, we can add filters. We would like to get all buildings between 10m^2 and 50m^2, so we will be using both the `is_greater_than` and `is_less_than` filter.\n",
    "\n",
    "When we add a filter, we can see that the number of returned features reduces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number features returned:  241\n"
     ]
    }
   ],
   "source": [
    "# create filters\n",
    "size_lower_bound = is_greater_than(\"SHAPE_Area\", 10)\n",
    "size_upper_bound = is_less_than(\"SHAPE_Area\", 50)\n",
    "# add filters to features object\n",
    "features.add_filters(size_lower_bound, size_upper_bound)\n",
    "\n",
    "results = features.query(limit=350)\n",
    "print(\"Number features returned: \", len(results[\"features\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If we want to add a third filter, we can simply create a new filter object and add to the FeaturesAPI object again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number features returned:  23\n"
     ]
    }
   ],
   "source": [
    "# create new filter\n",
    "length_lower_bound = is_greater_than(\"SHAPE_Length\", 30)\n",
    "# add filter to features object\n",
    "features.add_filters(length_lower_bound)\n",
    "\n",
    "results = features.query(limit=350)\n",
    "print(\"Number features returned: \", len(results[\"features\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "From this example, we have found how to create filters, how to apply them to a query, and how to use multiple filters at once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Example #2 - Topographic Area\n",
    "\n",
    "Now we have tried to create filters for a free product, we can turn our attention to premium data. Premium Ordnance Survey\n",
    "products have far more attributes that you can filter by, making this functionality even more powerful.\n",
    "\n",
    "Take the Topographic Area feature. There are 29 different attributes to filter by, including details about the area type\n",
    "and metadata about the feature's revision history. The available properties are:\n",
    "\n",
    "* OBJECTID\n",
    "* TOID\n",
    "* FeatureCode\n",
    "* Version\n",
    "* VersionDate\n",
    "* Theme\n",
    "* ThemeCount\n",
    "* CalculatedAreaValue\n",
    "* ChangeDate\n",
    "* ReasonForChange\n",
    "* ChangeHistoryCount\n",
    "* DescriptiveGroup\n",
    "* DescriptiveGroupCount\n",
    "* DescriptiveTerm\n",
    "* DescriptiveTermCount\n",
    "* Make\n",
    "* PhysicalLevel\n",
    "* PhysicalPresence\n",
    "* style_code\n",
    "* BHATopoAreaVersion\n",
    "* BHAProcessDate\n",
    "* AbsHMin\n",
    "* AbsH2\n",
    "* AbsHMax\n",
    "* RelH2\n",
    "* RelHMax\n",
    "* BHAConf\n",
    "* Shape_Length\n",
    "* Shape_Area\n",
    "\n",
    "With all this extra information, we can use even more filters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "crs = \"EPSG:27700\"\n",
    "# note: the product name in the python library isn't always exactly the same as the product name on the API\n",
    "# technical specification. In this case, the API's product is called Topography_TopographicArea\n",
    "# See all the products in feature_products.py to see if the name has changed.\n",
    "product = \"topographic_area\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Define an extent around Hammersmith, London\n",
    "W, S = (521202 , 177370)\n",
    "E, N = (523546 , 179409)\n",
    "extent = Extent.from_bbox((W, S, E, N), crs=crs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number features returned:  500\n"
     ]
    }
   ],
   "source": [
    "features = FeaturesAPI(key, product, extent)\n",
    "results = features.query(limit=500)\n",
    "print(\"Number features returned: \", len(results[\"features\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This time, as well as the size filter demonstrated in example 1, we will add a filter for text. We will be getting all buildings by adding an \"is_equal\" filter for the make of either \"Manmade\" and \"is_like\" for anything containing the word \"Park\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number features returned:  500\n"
     ]
    }
   ],
   "source": [
    "# add is_equal and is_like filter\n",
    "manmade = is_equal(\"Make\", \"Manmade\")\n",
    "park = is_like(\"Make\", \"*Park*\")\n",
    "size_lower_bound = is_greater_than(\"SHAPE_Area\", 30)\n",
    "size_upper_bound = is_less_than(\"SHAPE_Area\", 50)\n",
    "\n",
    "features.add_filters((manmade | park) & size_lower_bound & size_upper_bound)\n",
    "results = features.query(limit=500)\n",
    "print(\"Number features returned: \", len(results[\"features\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Thus, we can use filters for text to get in-depth results from Premium DataHub datasets.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}