Movielens-Life.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:08:58.662975Z",
     "start_time": "2017-03-04T14:08:58.658753+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "from datetime import datetime\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pprint import pprint\n",
    "\n",
    "np.set_printoptions(precision=3, suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Loading Data\n",
    "(Includes correct indexing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:09.042191Z",
     "start_time": "2017-03-04T14:08:59.529256+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "files = \"./ml-20m/\"\n",
    "# read files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Basic Preprocessing\n",
    "* splitting genres\n",
    "* annotating mean ratings\n",
    "* parsing timestmaps (slow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:09.063214Z",
     "start_time": "2017-03-04T14:09:09.043840+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# split genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:09.572397Z",
     "start_time": "2017-03-04T14:09:09.064510+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# extract R and annotate mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:38.296592Z",
     "start_time": "2017-03-04T14:09:09.573740+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# parse timestamps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Defining a similarity measure. (Pearson Correlation)\n",
    "If $a$, $b$ are two movies and $r_{m, u}$ is the rating that user $u$ gave movie $m$ then the similarity between $a$ and $b$ can be calculated using the pearson correlation:\n",
    "\n",
    "$$sim_{pearson}(a, b) = \\frac{\n",
    "    \\sum_{u\\in U}(r_{a,u} - \\overline{r_a}) (r_{b,u} - \\overline{r_b})\n",
    "}{\n",
    "    \\sqrt{\n",
    "        \\sum_{u\\in U_a}(r_{a,u} - \\overline{r_a})^2\n",
    "        \\sum_{u\\in U_b}(r_{b,u} - \\overline{r_b})^2\n",
    "    }\n",
    "}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:38.465898Z",
     "start_time": "2017-03-04T14:09:38.298371+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# single example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Common Base: Cosine\n",
    "\n",
    "$$\n",
    "sim_{cosine}(a,b) = cos(\\mathbf{r}_a, \\mathbf{r}_b) = \\frac{\n",
    "    \\mathbf{r}_a\\mathbf{r}_b\n",
    "}{\n",
    "    \\lVert\\mathbf{r}_a\\rVert_2 \\times \\lVert\\mathbf{r}_b\\rVert_2\n",
    "} = \\frac{\n",
    "    \\sum_{u\\in U_{a,b}}r_{a,u}r_{b,u}\n",
    "}{\n",
    "    \\sqrt{\n",
    "        \\sum_{u\\in U_a}r_{a,u}^2\n",
    "        \\sum_{u\\in U_b}r_{b,u}^2\n",
    "    }\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T13:09:38.591657Z",
     "start_time": "2017-03-04T14:09:38.467395+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def cosine(r1, r2):\n",
    "    pass\n",
    "\n",
    "cosine(R.ix[m1.name], R.ix[m2.name])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-07T17:08:23.415226Z",
     "start_time": "2017-03-07T18:08:23.364837+01:00"
    },
    "code_folding": [],
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sim_peason(m1, m2, r1, r2):\n",
    "    pass\n",
    "\n",
    "sim_peason(m1, m2, R.ix[m1.name], R.ix[m2.name])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T14:11:41.023666Z",
     "start_time": "2017-03-04T15:11:40.149623+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def calculate_sim(movieId, target_ids):\n",
    "    pass\n",
    "\n",
    "calculate_sim(10, movies.ix[:10-1].index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Parallelization with Spark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T14:15:55.808449Z",
     "start_time": "2017-03-04T15:15:55.792849+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# implement iterator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-04T14:25:17.872740Z",
     "start_time": "2017-03-04T15:25:17.770351+01:00"
    },
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# spark job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-03-05T00:31:45.891892Z",
     "start_time": "2017-03-04T15:25:33.332356+01:00"
    },
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# execute spark"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}