notebooks/01.1-explore-stew-geo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "159efddb-97fb-478b-9277-e3ab3d423faa",
   "metadata": {},
   "source": [
    "# Explore STEWMAP\n",
    "\n",
    "Multiple datasets have been provided for this project.  I am working with NYC2017_STEWMAP_Version2.gdb.\n",
    "\n",
    "This notebook will go through my standard analysis with a new dataset.  Specifically:\n",
    "\n",
    "  1. Read the data into a geodataframe\n",
    "  2. Check out the crs\n",
    "  3. Look at the attributes\n",
    "  4. Visualize\n",
    "  5. Attribute analysis - query, subset, ... for further processing\n",
    "  6. Office locations\n",
    "  7. Save STEWMAP parquet files\n",
    "  \n",
    "The \"output\" of this analysis is an understanding of the STEWMAP dataset that I can use to fuse with other NYC open datasets.\n",
    "  \n",
    "**Note-to-self:** Document/explain the env, startup, imports, ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "354f8eac-9985-4432-9339-5c6b5d916bab",
   "metadata": {},
   "source": [
    "# Read Data\n",
    "\n",
    "The dataset comes as a geodatabase.  First I want to see what layers are included.  Once I know that, I'll read a layer to create a geopandas geodataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d842fef4-ff2b-4a1b-8b11-e6880b7d2ca5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiona"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191aec30-4a81-4867-95c9-92e465942723",
   "metadata": {},
   "outputs": [],
   "source": [
    "fiona.listlayers('../data/raw/NYC-2017-STEW-MAP-Public-Version2/NYC2017_STEWMAP_Version2.gdb/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fbe2557-0c65-473d-884c-d12d1377eec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "office_locations_gdf = gpd.read_file('../data/raw/NYC-2017-STEW-MAP-Public-Version2/NYC2017_STEWMAP_Version2.gdb/',\n",
    "                           driver='FileGDB',\n",
    "                           layer=_[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "000c444b-0ef1-4450-ac96-6dbdc96cac69",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "turfs_gdf = gpd.read_file('../data/raw/NYC-2017-STEW-MAP-Public-Version2/NYC2017_STEWMAP_Version2.gdb/',\n",
    "                           driver='FileGDB',\n",
    "                           layer='NYC2017_STEWMAP_PublicPolygons_Version2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "753db995-fda9-441a-9f67-381ef2534428",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of office locations: {len(office_locations_gdf)}\")\n",
    "print(f\"Number of turfs: {len(turfs_gdf)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5386904a-cfc7-4813-accf-2ff1afcc8d1e",
   "metadata": {},
   "source": [
    "After office hour discussions, I am going to focus on the polygons (turfs).  Attributes are the same for each so ...  \n",
    "I have no idea why there are more office locations then turfs.  The answer to that question is for another day!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b775e7ed-4a6f-4d27-a47a-9f8c36c1634c",
   "metadata": {},
   "source": [
    "# CRS\n",
    "\n",
    "Since the ultimate objective is integration with other spatial datasets we need to understand the projection.  It is important for any spatial operations, i.e. joining, area/distance computations, ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db10742c-8a90-410f-a059-ae9ed6ee5729",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "turfs_gdf.crs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bcab6bd-2c7a-48ab-a24a-607253b8985d",
   "metadata": {},
   "source": [
    "Simple idiom to embed an iframe with external resource.\n",
    "\n",
    "Here's a description of the crs.  Note that the units for this projection are meters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2353ca0-e87d-4c14-9a54-ec1d1da651d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "IFrame(\"http://epsg.io/26918\", width=1200, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60139a3c-97bb-4082-b434-2b8746617a24",
   "metadata": {},
   "source": [
    "# Dataframe Info\n",
    "\n",
    "This section will look at the attributes.  \n",
    "\n",
    "At first look, I wonder why all those attributes are encoded as float64. Is it an esri thing? I don't usually worry about sizes but ...\n",
    "\n",
    "**Note:** Only looking at turfs for now.\n",
    "\n",
    "**Note-to-self:** Can you make this more efficient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80def2e5-28b4-4bfa-abb2-8428b91e5ae8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "turfs_gdf.info(verbose=True, show_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36332d62-1345-403b-8963-ea6f43677e4b",
   "metadata": {},
   "source": [
    "Lot's-o-variables here.  I wasn't sure how to deal with them at first.  During office hours Lindsay suggested I look at the [survey](https://www.nrs.fs.fed.us/STEW-MAP/resources/downloads/STEW-MAP%20NYC%202017%20Survey.pdf).  That definitely helped my understanding!\n",
    "\n",
    "Check out the dtypes.  I see two attributes with int64 dtype.  The rest are either strings (object) or floats.  Will probably make sense to revisit and remap these dtypes.\n",
    "\n",
    "**Note:** A slight digression, but something I realized further in the process.  According to the data dictionary, PopID is the unique ID.  Is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18b1d607-c978-4a0d-9c27-844b96051fe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.PopID.is_unique"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33171815-4419-43e3-a64b-3563b28ff1e8",
   "metadata": {},
   "source": [
    "Hummm..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aec69feb-68d4-4272-9617-41324c09e1e8",
   "metadata": {},
   "source": [
    "# Visualize\n",
    "\n",
    "Finally, I like to see what's in the dataset.  This is not meant to be detailed mapping, just to get situated.  Let's look at two visualizations:\n",
    "\n",
    "  1. the .plot function (matplotlib based)\n",
    "  2. .explore method that is folium based\n",
    "\n",
    "**Note-to-self:** I've created a subset but that doesn't seem to help.  Is it something about the geometries that slows it down?  Worth understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1788853c-cfe8-4d63-9255-d37ea1336a4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "explore_gdf = turfs_gdf[['OrgName', 'OrgWebSite', 'OrgTwitter', 'geometry']].reset_index().drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89d16660-7e34-4c0d-a666-76f33bb49173",
   "metadata": {},
   "source": [
    "Note I'm reprojecting so the axis labels make some sense (lat/long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e97611f1-cb52-4b3f-b458-746c7ffb656b",
   "metadata": {},
   "outputs": [],
   "source": [
    "explore_gdf.to_crs(\"epsg:4326\").plot(figsize=(18, 14));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24769ed5-2146-4eb4-bfb4-031d16d6cd41",
   "metadata": {},
   "source": [
    "Now we can look at the polygons with a map background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91d36164-ebe4-40bb-a274-052c0c85e7b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "explore_gdf.explore(\n",
    "    column='OrgName', \n",
    "    tooltip=['OrgName', 'OrgWebSite', 'OrgTwitter'],\n",
    "    popup=True, \n",
    "    legend=False,\n",
    "    width=1000,\n",
    "    tiles=\"CartoDB positron\", \n",
    "    cmap=\"Set1\", \n",
    "    style_kwds=dict(color=\"black\") \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae128e2b-9a6b-4afa-bceb-43b47c1adf98",
   "metadata": {},
   "source": [
    "# Attribute Analysis\n",
    "\n",
    "This last section will explore the attributes.  Remember the variables are best understood via the [survey](https://www.nrs.fs.fed.us/STEW-MAP/resources/downloads/STEW-MAP%20NYC%202017%20Survey.pdf).  Ultimately I need to develop \"tools\" to help fuse with other nyc datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a8d58d4-6eb7-4715-a25c-40485c1e8c21",
   "metadata": {},
   "source": [
    "The first attribute is the `primary site type`, the one site type you work at the most.\n",
    "\n",
    "I'll use the value_counts method to simply get the count of unique values.  You can plot as bar chart, but I just look at the numbers.  You can also normalize the count if you want percentages of total unique values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e6801e-6a55-4cbe-ab1e-525ecd56137d",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.PrimST.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85c0e2b6-7447-4c15-9fb5-22aa41e1c1bc",
   "metadata": {},
   "source": [
    "Now we can look at `Primary Focus`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb5414a3-66b6-44d0-8843-eb94139ff8be",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.PrimFocus.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa23a67-a07a-425a-bbba-f007bc15b1f4",
   "metadata": {},
   "source": [
    "One observation on the data are the fields with no value but they have a count.  Of course this type of thing happens in the real world.  You need to decide how to deal with it (infer, delete, exclude, ...)\n",
    "\n",
    "Being curious I tried something -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb912e45-a3d8-4831-ba8f-74140f153474",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(turfs_gdf.query(f\"PrimST == ' '\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "778d28c1-87fb-47e7-bd72-e3b2812de107",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(turfs_gdf.query(f\"PrimFocus == ' '\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "535435c2-cfd3-47f9-bbf6-771cd1dcbb5c",
   "metadata": {},
   "source": [
    "So it seems like the `missing` value is just a blank character.  This might be important later, we'll defer for now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b604c7c-e406-4334-9ba0-75ce4248816b",
   "metadata": {},
   "source": [
    "Next I'll look at the organizations by state, zip, and city.\n",
    "\n",
    "Remember I'm trying to understand how to build subsets of the data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a141f7b-7188-4636-87ad-eaeed6610a18",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.OrgState.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11ab3d13-f34c-4be2-ba27-3d9213e74885",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.OrgZip.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "225d8a53-b825-4e56-8d37-cf3b0c19ba72",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.OrgCity.value_counts()[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee0baee-9c45-4c60-8e32-8cb4a67dc391",
   "metadata": {},
   "source": [
    "Now I'd like to start drilling into the data a bit.\n",
    "\n",
    "For example the most selected PrimFocus was Environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ba444a4-159e-4f6a-9a74-d07a423110ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "environment_turfs_gdf = turfs_gdf.query(f\"PrimFocus == 'Environment'\").reset_index().drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf46b63f-526a-489c-9440-74b0d0a732c2",
   "metadata": {},
   "source": [
    "What is the distribution of `primary site types` for this focus area?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76b4e88d-8d20-404d-9ef9-180319754843",
   "metadata": {},
   "outputs": [],
   "source": [
    "environment_turfs_gdf.PrimST.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4af20ca-7338-4dda-b505-f0b9af89991d",
   "metadata": {},
   "outputs": [],
   "source": [
    "environment_turfs_gdf[['OrgName', 'OrgWebSite', 'OrgTwitter', 'geometry']].explore(\n",
    "    column='OrgName', \n",
    "    tooltip=['OrgName', 'OrgWebSite', 'OrgTwitter'],\n",
    "    popup=True, \n",
    "    legend=False,\n",
    "    width=1000,\n",
    "    tiles=\"CartoDB positron\", \n",
    "    cmap=\"Set1\", \n",
    "    style_kwds=dict(color=\"black\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7524f6ac-e492-41b5-8686-32dc8273ee29",
   "metadata": {},
   "outputs": [],
   "source": [
    "community_garden_turfs_gdf = turfs_gdf.query(f\"PrimST == 'Community Garden'\").reset_index().drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b002ce4-0859-4be2-a74d-19b358b72a8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "community_garden_turfs_gdf.PrimFocus.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "476ca9fb-d34e-4ce7-97d4-4bf31c7bde1e",
   "metadata": {},
   "source": [
    "Another way to look at the turfs is by city (OrgCity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "829f36f9-d1a2-4058-9337-27373eeb19a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "staten_island_gdf = turfs_gdf.query(f\"OrgCity == 'Staten Island'\").reset_index().drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ae2984d-986b-4392-b118-72481f95a2f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "staten_island_gdf[['OrgName', 'OrgWebSite', 'OrgTwitter', 'geometry']].explore(\n",
    "    column='OrgName', \n",
    "    tooltip=['OrgName', 'OrgWebSite', 'OrgTwitter'],\n",
    "    popup=True, \n",
    "    legend=False,\n",
    "    width=1000,\n",
    "    tiles=\"CartoDB positron\", \n",
    "    cmap=\"Set1\", \n",
    "    style_kwds=dict(color=\"black\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29306b1b-4c95-4dca-b65d-92dcd3f4074f",
   "metadata": {},
   "source": [
    "Next we'll look at our first example `combining` STEWMAP with other data sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "508ec7e3-ec2f-4b74-b217-248b5e4b6fbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "waterfront_turfs_gdf = turfs_gdf.query(f\"PrimST == 'Waterfront / Beach / Shoreline'\").reset_index().drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb8f683a-1f4a-4fc3-ac49-00ce0e0bb0e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "waterfront_turfs_gdf[['OrgName', 'OrgWebSite', 'OrgTwitter', 'geometry']].explore(\n",
    "    column='OrgName', \n",
    "    tooltip=['OrgName', 'OrgWebSite', 'OrgTwitter'],\n",
    "    popup=True, \n",
    "    legend=False,\n",
    "    width=1000,\n",
    "    tiles=\"CartoDB positron\", \n",
    "    cmap=\"Set1\", \n",
    "    style_kwds=dict(color=\"black\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5767bde4-145d-487f-9543-8b6437630c87",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"https://www1.nyc.gov/site/planning/data-maps/open-data/dwn-wrp.page\", width=1200, height=800)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7905ef8d-12a1-4e15-ae8e-0353acb6a086",
   "metadata": {},
   "outputs": [],
   "source": [
    "#fiona.listlayers('../data/NYCFutureHighTideWithSLR.gdb/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6667b611-cbf3-4051-9bea-0471babfe559",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastal_zone_gdf = gpd.read_file('../data/raw/NYCWRP_Shapefiles_2016/nycwrpsnwa_201601.shp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b25d2f10-c74a-468d-ba3b-da419736073b",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastal_zone_gdf.explore()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52cda1ca-4bda-4d33-ab66-c0be3f959bcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_output = Output(layout={'border': '1px solid black',\n",
    "                            'width': '50%'})\n",
    "\n",
    "density_output = Output(layout={'border': '1px solid black',\n",
    "                            'width': '50%'})\n",
    "\n",
    "with count_output:\n",
    "    display(staten_island_gdf[['OrgName', 'OrgWebSite', 'OrgTwitter', 'geometry']].explore())\n",
    "\n",
    "with density_output:\n",
    "    display(coastal_zone_gdf.explore())\n",
    "\n",
    "#print('\\nMaps for request type: ' + request_type + '\\n\\n')\n",
    "HBox([count_output, density_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be453071-242a-4728-8549-6936320caa57",
   "metadata": {},
   "source": [
    "# Office locations revisited\n",
    "\n",
    "To wrap up the analysis I want to examine the office locations.\n",
    "\n",
    "A straight mapping of the geometry (in my tools of choice) did not work.  Need to investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07dd198b-27af-4e75-bbdd-8609a705567b",
   "metadata": {},
   "outputs": [],
   "source": [
    "office_locations_gdf.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10402208-783a-4038-b7a1-6f9380384d25",
   "metadata": {},
   "outputs": [],
   "source": [
    "office_locations_gdf.iloc[27]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79e75aa9-e1cb-465f-b32c-e264d8d9e26d",
   "metadata": {},
   "source": [
    "Notice the geometry is POINT Z, i.e. it contains elevation.  They all seem to be 0 so it doesn't look like it is used.  I am going to build the xy version for further analysis (possibly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1368c02-471c-4c24-8273-137c4243ff85",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_geometry = [Point(x, y) for x, y in zip(office_locations_gdf.POINT_X, office_locations_gdf.POINT_Y)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8c3aa8c-56e7-4c25-8c89-fda7810af9e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_geometry[0].wkt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13671302-4677-4c5c-80e9-3d1c94f26dd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "office_locations_gdf['geometry'] = new_geometry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f48c00ef-23f1-401e-9f5f-b2555835ca86",
   "metadata": {},
   "outputs": [],
   "source": [
    "office_locations_gdf.explore()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9578f7b1-02cd-4b18-b49d-8ca198086ceb",
   "metadata": {},
   "source": [
    "# Save STEWMAP parquet\n",
    "\n",
    "I am going to save the data as parquet files for upstream analysis.  I really should deal with the PopID here, but will wait until later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "122f4d80-5561-4e4c-b096-25ff6275679f",
   "metadata": {},
   "outputs": [],
   "source": [
    "office_locations_gdf.to_parquet('../data/processed/office-locations.parq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36a676b9-0026-4078-8b2f-3a37f1dddefc",
   "metadata": {},
   "outputs": [],
   "source": [
    "turfs_gdf.to_parquet('../data/processed/turfs.parq')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4649a11b-5a65-4c19-84f7-dd97ac361665",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}