5. Loading and Visualizing Network Data (Student).ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import networkx as nx\n",
    "import os\n",
    "import numpy as np\n",
    "import warnings\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from nxviz import CircosPlot\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tables to Networks, Networks to Tables\n",
    "\n",
    "Networks can be represented in a tabular form in two ways: As an adjacency list with edge attributes stored as columnar values, and as a node list with node attributes stored as columnar values.\n",
    "\n",
    "Storing the network data as a single massive adjacency table, with node attributes repeated on each row, can get unwieldy, especially if the graph is large, or grows to be so. One way to get around this is to store two files: one with node data and node attributes, and one with edge data and edge attributes. \n",
    "\n",
    "The Divvy bike sharing dataset is one such example of a network data set that has been stored as such."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Loading Node Lists and Adjacency Lists\n",
    "\n",
    "Let's use the Divvy bike sharing data set as a starting point. The Divvy data set is comprised of the following data:\n",
    "\n",
    "- Stations and metadata (like a node list with attributes saved)\n",
    "- Trips and metadata (like an edge list with attributes saved)\n",
    "\n",
    "The `README.txt` file in the Divvy directory should help orient you around the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# This block of code checks to make sure that a particular directory is present.\n",
    "if \"divvy_2013\" not in os.listdir('datasets/'):\n",
    "    print('Unzip the divvy_2013.zip file in the datasets folder.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "stations = pd.read_csv('datasets/divvy_2013/Divvy_Stations_2013.csv', parse_dates=['online date'], index_col='id', encoding='utf-8')\n",
    "stations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "trips = pd.read_csv('datasets/divvy_2013/Divvy_Trips_2013.csv', \n",
    "                    parse_dates=['starttime', 'stoptime'], \n",
    "                    index_col=['trip_id'])\n",
    "trips = trips.sort()\n",
    "trips"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "At this point, we have our `stations` and `trips` data loaded into memory. \n",
    "\n",
    "How we construct the graph depends on the kind of questions we want to answer, which makes the definition of the \"unit of consideration\" (or the entities for which we are trying to model their relationships) is extremely important. \n",
    "\n",
    "Let's try to answer the question: \"What are the most popular trip paths?\" In this case, the bike station is a reasonable \"unit of consideration\", so we will use the bike stations as the nodes. \n",
    "\n",
    "To start, let's initialize an directed graph `G`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "G = nx.DiGraph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Then, let's iterate over the `stations` DataFrame, and add in the node attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for r, d in stations.iterrows(): # call the pandas DataFrame row-by-row iterator\n",
    "    G.add_node(r,   attr_dict=d.to_dict()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In order to answer the question of \"which stations are important\", we need to specify things a bit more. Perhaps a measure such as **betweenness centrality** or **degree centrality** may be appropriate here.\n",
    "\n",
    "The naive way would be to iterate over all the rows. Go ahead and try it at your own risk - it may take a long time :-). Alternatively, I would suggest doing a `pandas` `groupby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# # Run the following code at your own risk :)\n",
    "# for r, d in trips.iterrows():\n",
    "#     start = d['from_station_id']\n",
    "#     end = d['to_station_id']\n",
    "#     if (start, end) not in G.edges():\n",
    "#         G.add_edge(start, end, count=1)\n",
    "#     else:\n",
    "#         G.edge[start][end]['count'] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for (start, stop), d in trips.groupby(['from_station_id', 'to_station_id']):\n",
    "    G.add_edge(start, stop, count=len(d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Flex your memory muscles: can you make a scatter plot of the distribution of the number edges that have a certain number of trips? \n",
    "\n",
    "The key should be the number of trips between two nodes, and the value should be the number of edges that have that number of trips."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "# Count the number of edges that have x trips recorded on them.\n",
    "trip_count_distr = ______________________________\n",
    "\n",
    "# Then plot the distribution of these\n",
    "plt.scatter(_______________, _______________, alpha=0.1)\n",
    "plt.yscale('log')\n",
    "plt.xlabel('num. of trips')\n",
    "plt.ylabel('num. of edges')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Create a new graph, and filter out the edges such that only those with more than 100 trips taken (i.e. `count >= 100`) are left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Filter the edges to just those with more than 100 trips.\n",
    "G_filtered = G.copy()\n",
    "for u, v, d in G.edges(data=True):\n",
    "    # Fill in your code here.\n",
    "        \n",
    "len(G_filtered.edges())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's now try drawing the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Use `nx.draw(my_graph)` to draw the filtered graph to screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in your code here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Try visualizing the graph using a CircosPlot. Order the nodes by their connectivity in the **original** graph, but plot only the **filtered** graph edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for n in G_filtered.nodes():\n",
    "    ____________\n",
    "\n",
    "c = CircosPlot(__________)\n",
    "__________\n",
    "plt.savefig('images/divvy.png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In this visual, nodes are sorted from highest connectivity to lowest connectivity in the **unfiltered** graph.\n",
    "\n",
    "Edges represent only trips that were taken >100 times between those two nodes.\n",
    "\n",
    "Some things should be quite evident here. There are lots of trips between the highly connected nodes and other nodes, but there are local \"high traffic\" connections between stations of low connectivity as well (nodes in the top-right quadrant)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Saving NetworkX Graph Files\n",
    "\n",
    "NetworkX's API offers many formats for storing graphs to disk. If you intend to work exclusively with NetworkX, then pickling the file to disk is probably the easiest way.\n",
    "\n",
    "To write to disk: \n",
    "\n",
    "    nx.write_gpickle(G, handle)\n",
    "\n",
    "To load from disk:\n",
    "    \n",
    "    G = nx.read_gpickle(handle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "nx.write_gpickle(G, 'datasets/divvy_2013/divvy_graph.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "G = nx.read_gpickle('datasets/divvy_2013/divvy_graph.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Environment (conda_nams)",
   "language": "python",
   "name": "conda_nams"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}