notebooks/05.5-clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "18694c25-1597-43b4-80e2-2ea54c6121ca",
   "metadata": {},
   "source": [
    "# Poor Man's Hot Spot\n",
    "\n",
    "I'm trying to understand what is going on with the Sewer 311 requests.  Before I look at the standard ways to build hot spots (DBSCAN, Getis-Ord Gi*, ...) I need to gain some intuition into what is going on.\n",
    "\n",
    "This notebook is also used to relate DEM with 311 locations.  Maybe this should be moved?\n",
    "\n",
    "\n",
    "**Note:** This nb is computationaly expensive because of the DEM + point integration.  Need to understand why and fix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01cd068-2854-4880-a721-b669096ced0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_311_gdf = gpd.read_parquet('../data/processed/brooklyn/brooklyn-2021-311.parq').reset_index().drop(columns='index')\n",
    "\n",
    "water_sewer_311_gdf = brooklyn_311_gdf[brooklyn_311_gdf['Complaint Type'].isin(['Water System', 'Sewer'])]\n",
    "\n",
    "sewer_311_gdf = water_sewer_311_gdf.query(f\"`Complaint Type` == 'Sewer'\").reset_index().drop(columns='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87554b98-6dd7-4ea9-a747-5e42e507ed66",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_gdf = gpd.read_parquet('../data/processed/admin-boundaries/brooklyn.parq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7e1b0c5-487d-48ca-9e33-9e9d1debfedf",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_turfs_gdf = gpd.read_parquet('../data/processed/brooklyn/brooklyn-turfs.parq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e25ce41-e78d-4854-9d01-831f54e0c1ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_turfs_subset_gdf = brooklyn_turfs_gdf[['OrgName', 'OrgWebSite', 'PrimST', 'PopID', 'Shape_Area', 'geometry']].copy().to_crs('epsg:2263')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca871dfd-6c87-4a99-8687-54cba5a35fc3",
   "metadata": {},
   "source": [
    "pertinent sounds awkward?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "316fdf5b-8b74-4e7c-aede-372c076943be",
   "metadata": {},
   "outputs": [],
   "source": [
    "pertinent_turfs_primst = ['Waterfront / Beach / Shoreline', \n",
    "                          'Watershed / Sewershed', \n",
    "                          'Stream / River / Canal', \n",
    "                          'Salt Marsh', \n",
    "                          'Public Right of Way (Sidewalk, street ends, traffic island, public plaza)', \n",
    "                          'Freshwater Wetland']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e176c06a-f6c4-4fc0-a9c5-4d91d5599c49",
   "metadata": {},
   "source": [
    "Maybe this one could be renamed to primst_based_turfs_gdf?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c69fcc0-fc02-4ffa-a03a-fae897967308",
   "metadata": {},
   "outputs": [],
   "source": [
    "pertinent_turfs_gdf = brooklyn_turfs_subset_gdf.query(f\"PrimST in @pertinent_turfs_primst\").reset_index().drop(columns='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e20bf514-874c-412e-b433-9a11a0267013",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pertinent_turfs_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3eb41d2c-6f93-4e31-997e-531ad0ca3753",
   "metadata": {},
   "outputs": [],
   "source": [
    "pertinent_turfs_gdf.sort_values('Shape_Area', inplace=True, ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1da0ecd-77d4-4185-a9a3-b0c7a77babed",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyleaflet import Heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e87dc06e-70fd-42b5-9e97-be694b78b745",
   "metadata": {},
   "outputs": [],
   "source": [
    "center = brooklyn_gdf.iloc[0].geometry.centroid.y, brooklyn_gdf.iloc[0].geometry.centroid.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01e04b89-2cc9-460f-a138-867e397277df",
   "metadata": {},
   "outputs": [],
   "source": [
    "imagery = basemap_to_tiles(basemaps.Esri.WorldImagery)\n",
    "imagery.base = True\n",
    "osm = basemap_to_tiles(basemaps.OpenStreetMap.Mapnik)\n",
    "osm.base = True\n",
    "\n",
    "google_map = TileLayer(\n",
    "    url=\"https://mt1.google.com/vt/lyrs=m&x={x}&y={y}&z={z}\",\n",
    "    attribution=\"Google\",\n",
    "    name=\"Google Maps\",\n",
    ")\n",
    "google_map.base = True\n",
    "\n",
    "google_satellite = TileLayer(\n",
    "    url=\"https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}\",\n",
    "    attribution=\"Google\",\n",
    "    name=\"Google Satellite\"\n",
    ")\n",
    "google_satellite.base = True\n",
    "\n",
    "map_display = Map(center=center, zoom=12,\n",
    "                  layers=[google_satellite, google_map, imagery, osm],\n",
    "                  layout=Layout(height=\"900px\"),\n",
    "                  scroll_wheel_zoom=True)\n",
    "\n",
    "map_display.add_control(LayersControl())\n",
    "\n",
    "map_display.add_control(FullScreenControl())\n",
    "\n",
    "map_display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f635874-a55e-4d5f-b3c1-6f9c51f631c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_data = [[point.xy[1][0], point.xy[0][0]] for point in sewer_311_gdf.geometry]\n",
    "heat_map = Heatmap(locations=heat_data, radius=20, blur=10, name='Heat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0706b71-1650-4da8-905f-ba67fc61eed3",
   "metadata": {},
   "outputs": [],
   "source": [
    "map_display.add_layer(heat_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df4e02cb-94b2-49bb-a52e-f3a00e58e487",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_color(feature):\n",
    "    return {\n",
    "        'color': 'black',\n",
    "        'fillColor': random.choice(['red', 'yellow', 'green', 'orange', 'purple', 'blue']),\n",
    "    }\n",
    "\n",
    "import json\n",
    "import random\n",
    "#turfs_geojson = brooklyn_turf_subset_gdf.to_json()\n",
    "turfs_geojson = pertinent_turfs_gdf.iloc[25:55].to_crs('epsg:4326').to_json()\n",
    "\n",
    "geo_json = GeoJSON(\n",
    "    data=json.loads(turfs_geojson),\n",
    "    #style={\n",
    "    #    'opacity': 0.1, 'dashArray': '9', 'fillOpacity': 0.1, 'weight': 1\n",
    "   # },\n",
    "    hover_style={\n",
    "        'color': 'white', 'dashArray': '0', 'fillOpacity': 0.8\n",
    "    },\n",
    "    #style_callback=random_color,\n",
    "    name='turfs geojson'\n",
    ")\n",
    "\n",
    "map_display.add_layer(geo_json)\n",
    "\n",
    "turf_html = HTML('''Hover over a turf''')\n",
    "turf_html.layout.margin = '0px 20px 20px 20 px'\n",
    "turf_control = WidgetControl(widget=turf_html, position='bottomright')\n",
    "\n",
    "def update_turf_html(feature, **kwargs):\n",
    "    turf_html.value = f\"<b>Name: {feature['properties']['OrgName']}<br><b>Primary: {feature['properties']['PrimST']}\"\n",
    "    \n",
    "map_display.add_control(turf_control)  # does += work for this?\n",
    "\n",
    "geo_json.on_hover(update_turf_html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ac8452e-3c4a-48d2-9556-abf416f53321",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(sewer_311_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa1620d0-0736-4dcd-9d30-1255a42f140a",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(water_sewer_311_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70b00a9c-afd9-42e2-9022-d03c303783d2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19dae1ef-7fb2-4e51-9ce7-733bd588381c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#call_density_gdf = GeoDataFrame(sewer_311_gdf.geometry.value_counts().to_frame().reset_index().rename(columns={'index': 'geometry', 'geometry': 'count'}))\n",
    "call_density_gdf = GeoDataFrame(water_sewer_311_gdf.geometry.value_counts().to_frame().reset_index().rename(columns={'index': 'geometry', 'geometry': 'count'}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f82074-b21a-494d-b534-88d4a484bd0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_two_gdf = call_density_gdf[call_density_gdf['count'] > 2].reset_index().drop(columns='index')\n",
    "more_than_three_gdf = call_density_gdf[call_density_gdf['count'] > 3].reset_index().drop(columns='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd6b01b5-1c8d-4cc6-ab77-e110390bbf5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_two_gdf['count'].value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "624de210-068f-429d-a9d9-c29f6b361e12",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['count'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab0412f-eb49-46da-b0cf-8516fc7c749d",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_two_gdf['bin'] = pd.cut(more_than_two_gdf['count'], bins=[0, 10, 22, 42])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e5d4df9-3e59-45f4-b500-7c055ce8f3ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_two_gdf['bin'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef8632d-d6af-4bb6-b1ad-401814407c20",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['bin'] = pd.cut(more_than_three_gdf['count'], bins=[0, 10, 22, 42], labels=['small', 'medium', 'large'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49449fe3-c553-4b91-9944-1c80b1246eb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['bin'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f7966da-88f0-4945-bcfa-2b693ce191c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "c_map = {'small': '#4E9A26', 'medium': '#EBC621', 'large': '#AC1212'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328271d6-c11b-45b2-aecb-abf19400c7e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "markers = list()\n",
    "\n",
    "for i, row in tqdm(more_than_three_gdf.iterrows()):\n",
    "    \n",
    "    fill_color = c_map[row.bin]\n",
    "    marker = CircleMarker(location=(row.geometry.y, row.geometry.x), radius=5, stroke=False, fill_color=fill_color, fill_opacity=1.0)\n",
    "    msg = HTML()\n",
    "    msg.value = \"count: {}\".format(row['count'])\n",
    "    marker.popup = msg\n",
    "    markers.append(marker)\n",
    "    more_than_three_gdf.loc[i, 'marker'] = marker\n",
    "\n",
    "dep_cluster = MarkerCluster(markers=markers, name='311 Call')\n",
    "\n",
    "map_display.add_layer(LayerGroup(name=f\"311\", layers=markers))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "450b3a66-549a-4129-9897-c370c62b3b90",
   "metadata": {},
   "source": [
    "# DEM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b0c038c-2898-4307-8adc-d314269a376e",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_dem_gdf = gpd.read_parquet('../data/processed/brooklyn/brooklyn-dem.parq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1563d45a-d55e-46e1-9be7-5a3ade66c61d",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(brooklyn_dem_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9313c2e1-61fe-4377-8f40-0cb8e9b7ddb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "brooklyn_dem_gdf.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd12274f-ab0c-40cf-96fb-0697df2ebfcb",
   "metadata": {},
   "source": [
    "https://answerbun.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af4d334a-923e-4b21-8a4b-0d198b52ce37",
   "metadata": {},
   "source": [
    "**Note:** This is very computationally intense.  Beware. I saved a version for my purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faa6d231-a9da-47df-8a15-7a187c5c446c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapely.ops import nearest_points\n",
    "# unary union of the gpd2 geomtries \n",
    "pts3 = brooklyn_dem_gdf.geometry.unary_union\n",
    "def near(point, pts=pts3):\n",
    "     # find the nearest point and return the corresponding Place value\n",
    "     nearest = brooklyn_dem_gdf.geometry == nearest_points(point, pts)[1]\n",
    "     return brooklyn_dem_gdf[nearest].elevation.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b37b8898-ad8e-4d39-81db-c2c900459c85",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "more_than_three_gdf['elevation'] = more_than_three_gdf.apply(lambda row: near(row.geometry), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8e58455-cb20-4f08-8af1-c549388a2808",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf.iloc[27]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c8341b-3e7e-4589-85c2-84cc2820c3c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf.elevation.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69ae27bf-6eda-41f2-a1e9-6f5e4a576d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf[['geometry', 'count', 'bin', 'elevation']].to_parquet('../data/processed/brooklyn/brooklyn-311-water-sewer-elevation.parq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54f0690-fc82-44f4-ab9a-b3cfd13c7994",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['elevation'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0997fc1-0acb-433f-a06c-84f966f5add1",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['elevation'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88d97515-d585-42ca-a059-9b87a6ee6fb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "more_than_three_gdf['elevation'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b83d76b-7416-49ff-b1b1-8e6cdfc5b26c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}