k_Nearest_Neighbors-Accelerometer-Features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   This notebook shows an example of the k-NN algorithm applied to classification of time series data (accelerometer data). The task here is to classify 6 different types of activities (walking, walking upstairs, walking downstairs, sitting, standing, laying) based on x, y and z accelerometer signals. Instead of using the raw signals, we extract features from the signals (statistical measures, geometrical features and frequency domain features) and use them as input for the knn algorithm. We first use features from x, y and z axis of the accelerometer and then we only use features from x axis and compare the results to the results using raw signals and and [knn with Dynamic Time Warping (DTW) distance](https://github.com/jeandeducla/ML-Time-Series/blob/master/k_Nearest_Neighbors-Accelerometer-Raw.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using matplotlib backend: MacOSX\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pylab as plt \n",
    "from scipy.stats import mode\n",
    "from sklearn.metrics import classification_report, f1_score\n",
    "from sklearn import preprocessing\n",
    "import time\n",
    "import matplotlib.pylab as plt \n",
    "%matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the raw accelerometer signals of the 3 available axis (x, y and z). Each sample of the data set is a 2.56s window of an activity being performed recorded at a 50Hz rate which makes 128 readings per sample per axis.  \n",
    "We are going to consider the same subset of the training and test sets as the one we use in the [notebook](https://github.com/jeandeducla/ML-Time-Series/blob/master/k_Nearest_Neighbors-Accelerometer-Raw.ipynb) showing how to use knn with raw signals and Dynamic Time Warping distance in order to compare these two approaches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train_x_raw shape : (368, 128)\n",
      "X_test_x_raw shape : (148, 128)\n",
      "X_train_y_raw shape : (368, 128)\n",
      "X_test_y_raw shape : (148, 128)\n",
      "X_train_z_raw shape : (368, 128)\n",
      "X_test_z_raw shape : (148, 128)\n"
     ]
    }
   ],
   "source": [
    "os.chdir('data')\n",
    "\n",
    "n_skip = 20\n",
    "\n",
    "# Raw signals\n",
    "# X axis\n",
    "X_train_x_raw = np.loadtxt('X_x_train.txt')[::n_skip]\n",
    "X_test_x_raw = np.loadtxt('X_x_test.txt')[::n_skip]\n",
    "# Y axis\n",
    "X_train_y_raw = np.loadtxt('X_y_train.txt')[::n_skip]\n",
    "X_test_y_raw = np.loadtxt('X_y_test.txt')[::n_skip]\n",
    "# Z axis\n",
    "X_train_z_raw = np.loadtxt('X_z_train.txt')[::n_skip]\n",
    "X_test_z_raw = np.loadtxt('X_z_test.txt')[::n_skip]\n",
    "\n",
    "print(\"X_train_x_raw shape : {}\".format(X_train_x_raw.shape))\n",
    "print(\"X_test_x_raw shape : {}\".format(X_test_x_raw.shape))\n",
    "print(\"X_train_y_raw shape : {}\".format(X_train_y_raw.shape))\n",
    "print(\"X_test_y_raw shape : {}\".format(X_test_y_raw.shape))\n",
    "print(\"X_train_z_raw shape : {}\".format(X_train_z_raw.shape))\n",
    "print(\"X_test_z_raw shape : {}\".format(X_test_z_raw.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature extraction helper functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As explained at the beginning of this notebook, we are not going to use the raw signals as input for our model but we are going to extract features from the signals. The following functions help us build the feature vectors out of the raw signals we have loaded above.\n",
    "These functions help us extract statistical and geometrical features from raw signals and jerk signals (acceleration first derivative), frequency domain features from raw signals and jerk signals\n",
    "For each sample we extract the following features:\n",
    "  - **x,y and z raw signals** : mean, max, min, standard deviation, skewness, kurtosis, interquartile range, median absolute deviation, area under curve, area under squared curve\n",
    "  - **x,y and z jerk signals (first derivative)** : mean, max, min, standard deviation, skewness, kurtosis, interquartile range, median absolute deviation, area under curve, area under squared curve\n",
    "  - **x,y and z raw signals Discrete Fourrier Transform**: mean, max, min, standard deviation, skewness, kurtosis, interquartile range, median absolute deviation, area under curve, area under squared curve, weighted mean frequency, 5 first DFT coefficients, 5 first local maxima of DFT coefficients and their corresponding frequencies.\n",
    "  - **x,y and z jerk signals Discrete Fourrier Transform**: mean, max, min, standard deviation, skewness, kurtosis, interquartile range, median absolute deviation, area under curve, area under squared curve, weighted mean frequency, 5 first DFT coefficients, 5 first local maxima of DFT coefficients and their corresponding frequencies.\n",
    "  - **x,y and z correlation coefficients**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import scipy.stats as st\n",
    "from scipy.fftpack import fft, fftfreq \n",
    "from scipy.signal import argrelextrema\n",
    "import operator\n",
    "\n",
    "def stat_area_features(x, Te=1.0):\n",
    "\n",
    "    mean_ts = np.mean(x, axis=1).reshape(-1,1) # mean\n",
    "    max_ts = np.amax(x, axis=1).reshape(-1,1) # max\n",
    "    min_ts = np.amin(x, axis=1).reshape(-1,1) # min\n",
    "    std_ts = np.std(x, axis=1).reshape(-1,1) # std\n",
    "    skew_ts = st.skew(x, axis=1).reshape(-1,1) # skew\n",
    "    kurtosis_ts = st.kurtosis(x, axis=1).reshape(-1,1) # kurtosis \n",
    "    iqr_ts = st.iqr(x, axis=1).reshape(-1,1) # interquartile rante\n",
    "    mad_ts = np.median(np.sort(abs(x - np.median(x, axis=1).reshape(-1,1)),\n",
    "                               axis=1), axis=1).reshape(-1,1) # median absolute deviation\n",
    "    area_ts = np.trapz(x, axis=1, dx=Te).reshape(-1,1) # area under curve\n",
    "    sq_area_ts = np.trapz(x ** 2, axis=1, dx=Te).reshape(-1,1) # area under curve ** 2\n",
    "\n",
    "    return np.concatenate((mean_ts,max_ts,min_ts,std_ts,skew_ts,kurtosis_ts,\n",
    "                           iqr_ts,mad_ts,area_ts,sq_area_ts), axis=1)\n",
    "\n",
    "def frequency_domain_features(x, Te=1.0):\n",
    "\n",
    "    # As the DFT coefficients and their corresponding frequencies are symetrical arrays\n",
    "    # with respect to the middle of the array we need to know if the number of readings \n",
    "    # in x is even or odd to then split the arrays...\n",
    "    if x.shape[1]%2 == 0:\n",
    "        N = int(x.shape[1]/2)\n",
    "    else:\n",
    "        N = int(x.shape[1]/2) - 1\n",
    "    xf = np.repeat(fftfreq(x.shape[1],d=Te)[:N].reshape(1,-1), x.shape[0], axis=0) # frequencies\n",
    "    dft = np.abs(fft(x, axis=1))[:,:N] # DFT coefficients   \n",
    "    \n",
    "    # statistical and area features\n",
    "    dft_features = stat_area_features(dft, Te=1.0)\n",
    "    # weighted mean frequency\n",
    "    dft_weighted_mean_f = np.average(xf, axis=1, weights=dft).reshape(-1,1)\n",
    "    # 5 first DFT coefficients \n",
    "    dft_first_coef = dft[:,:5]    \n",
    "    # 5 first local maxima of DFT coefficients and their corresponding frequencies\n",
    "    dft_max_coef = np.zeros((x.shape[0],5))\n",
    "    dft_max_coef_f = np.zeros((x.shape[0],5))\n",
    "    for row in range(x.shape[0]):\n",
    "        # finds all local maximas indexes\n",
    "        extrema_ind = argrelextrema(dft[row,:], np.greater, axis=0) \n",
    "        # makes a list of tuples (DFT_i, f_i) of all the local maxima\n",
    "        # and keeps the 5 biggest...\n",
    "        extrema_row = sorted([(dft[row,:][j],xf[row,j]) for j in extrema_ind[0]],\n",
    "                             key=operator.itemgetter(0), reverse=True)[:5] \n",
    "        for i, ext in enumerate(extrema_row):\n",
    "            dft_max_coef[row,i] = ext[0]\n",
    "            dft_max_coef_f[row,i] = ext[1]    \n",
    "    \n",
    "    return np.concatenate((dft_features,dft_weighted_mean_f,dft_first_coef,\n",
    "                           dft_max_coef,dft_max_coef_f), axis=1)\n",
    "\n",
    "def make_feature_vector(x,y,z, Te=1.0):\n",
    "\n",
    "    # Raw signals :  stat and area features\n",
    "    features_xt = stat_area_features(x, Te=Te)\n",
    "    features_yt = stat_area_features(y, Te=Te)\n",
    "    features_zt = stat_area_features(z, Te=Te)\n",
    "    \n",
    "    # Jerk signals :  stat and area features\n",
    "    features_xt_jerk = stat_area_features((x[:,1:]-x[:,:-1])/Te, Te=Te)\n",
    "    features_yt_jerk = stat_area_features((y[:,1:]-y[:,:-1])/Te, Te=Te)\n",
    "    features_zt_jerk = stat_area_features((z[:,1:]-z[:,:-1])/Te, Te=Te) \n",
    "    \n",
    "    # Raw signals : frequency domain features \n",
    "    features_xf = frequency_domain_features(x, Te=1/Te)\n",
    "    features_yf = frequency_domain_features(y, Te=1/Te)\n",
    "    features_zf = frequency_domain_features(z, Te=1/Te)\n",
    "    \n",
    "    # Jerk signals : frequency domain features \n",
    "    features_xf_jerk = frequency_domain_features((x[:,1:]-x[:,:-1])/Te, Te=1/Te)\n",
    "    features_yf_jerk = frequency_domain_features((y[:,1:]-y[:,:-1])/Te, Te=1/Te)\n",
    "    features_zf_jerk = frequency_domain_features((z[:,1:]-z[:,:-1])/Te, Te=1/Te)\n",
    "    \n",
    "    # Raw signals correlation coefficient between axis\n",
    "    cor = np.empty((x.shape[0],3))\n",
    "    for row in range(x.shape[0]):\n",
    "        xyz_matrix = np.concatenate((x[row,:].reshape(1,-1),y[row,:].reshape(1,-1),\n",
    "                                     z[row,:].reshape(1,-1)), axis=0)\n",
    "        cor[row,0] = np.corrcoef(xyz_matrix)[0,1]\n",
    "        cor[row,1] = np.corrcoef(xyz_matrix)[0,2]\n",
    "        cor[row,2] = np.corrcoef(xyz_matrix)[1,2]\n",
    "    \n",
    "    return np.concatenate((features_xt, features_yt, features_zt,\n",
    "                           features_xt_jerk, features_yt_jerk, features_zt_jerk,\n",
    "                           features_xf, features_yf, features_zf,\n",
    "                           features_xf_jerk, features_yf_jerk, features_zf_jerk,\n",
    "                           cor), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape : (368, 219)\n",
      "X_test shape: (148, 219)\n"
     ]
    }
   ],
   "source": [
    "X_train = make_feature_vector(X_train_x_raw, X_train_y_raw, X_train_z_raw, Te=1/50)\n",
    "X_test = make_feature_vector(X_test_x_raw, X_test_y_raw, X_test_z_raw, Te=1/50)\n",
    "\n",
    "print(\"X_train shape : {}\".format(X_train.shape))\n",
    "print(\"X_test shape: {}\".format(X_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We scale the features (standard scaler i.e. each feature column has a zero mean and one standard deviation). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler = preprocessing.StandardScaler().fit(X_train)\n",
    "X_train = scaler.transform(X_train) \n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the label vectors..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train shape : (368,)\n",
      "y_test shape : (148,)\n"
     ]
    }
   ],
   "source": [
    "y_train = np.loadtxt('y_train.txt')[::n_skip]\n",
    "y_test = np.loadtxt('y_test.txt')[::n_skip]\n",
    "\n",
    "print(\"y_train shape : {}\".format(y_train.shape))\n",
    "print(\"y_test shape : {}\".format(y_test.shape))\n",
    "\n",
    "label_names = ['Walking', 'Walking upstairs', 'Walking downstairs', 'Sitting', 'Standing', 'Laying']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-NN algorithm "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the three following functions we implement a brute force knn algorithm using euclidian distance: the first function returns the euclidian distance between two time series, the second function returns the distance matrix between samples from X_train and X_test and finally the last function returns the predicted labels for X_test samples according to the distance matrix and paramater k."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def euclidian_distance(x1,x2):\n",
    "    return np.linalg.norm(x1-x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def make_distance_matrix(X_train, X_test, w=60, distance = euclidian_distance):\n",
    "    \"\"\" This function returns the distance matrix between samples of X_train and X_tes according to a \n",
    "    similarity measure.\n",
    "    INPUTS:\n",
    "        - X_train a (n, p) numpy array with n:number of training samples and m: number of features\n",
    "        - X_test a (m, p) numpy array with m: number of test samples and m as above\n",
    "        - w DTW window\n",
    "        - distance_type the type of distance to consider for the algorithm ['euclidian', 'DTW']\n",
    "    OUTPUTS:\n",
    "        - dis_m a (m,n) numpy array with dist_m[i,j] = distance(X_test[i,:], X_train[j,:])\n",
    "    \"\"\"\n",
    "    \n",
    "    # Distance matrix calculation\n",
    "    n = X_train.shape[0]\n",
    "    m = X_test.shape[0]  \n",
    "    dist_m = np.zeros((m,n))\n",
    "    for row, test_spl in enumerate(X_test):\n",
    "        for col, train_spl in enumerate(X_train):\n",
    "            if distance == euclidian_distance:\n",
    "                dist_row_col = distance(test_spl, train_spl)\n",
    "                dist_m[row,col] = dist_row_col\n",
    "            else:\n",
    "                dist_row_col = distance(test_spl, train_spl, w)\n",
    "                dist_m[row,col] = dist_row_col                    \n",
    "    return dist_m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_k_closest(dist_m, y_train, k):\n",
    "    \"\"\" This function returns the most represented label among the k nearest neighbors of each sample from\n",
    "    X_test.\n",
    "    INPUTS:\n",
    "        - dist_m a (m,n) numpy array with dist_m[i,j] = distance(X_test[i,:], X_train[j,:])\n",
    "        - y_train a (n,) numpy array with X_train labels\n",
    "        - k number of neighbors to consider (int)\n",
    "    OUPUTS:\n",
    "        - y_pred a (m,) numpy array of predicted labels for X_test\n",
    "    \"\"\"\n",
    "    knn_indexes = np.argsort(dist_m)[:,:k]\n",
    "    knn_labels = y_train[knn_indexes]\n",
    "    y_pred = mode(knn_labels, axis=1)[0]\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## kNN with euclidian distance and x, y and z axis feature vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first compute the distance matrix between features vectors of X_train and X_test. Note that the execution time is under a second (compared to around 20 min for knn with DTW)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Execution time: 0.00 min 0.54 s \n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "dist_m = make_distance_matrix(X_train, X_test)\n",
    "stop = time.time()\n",
    "\n",
    "print(\"Execution time: {:.2f} min {:.2f} s \".format((stop-start) // 60, (stop-start) % 60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we find the k closest neighbors with k = 1..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters:\n",
      "k = 1\n",
      "\n",
      "\n",
      "Test set report\n",
      "                    precision    recall  f1-score   support\n",
      "\n",
      "           Walking       0.61      0.96      0.75        23\n",
      "  Walking upstairs       0.88      0.56      0.68        25\n",
      "Walking downstairs       0.84      0.70      0.76        23\n",
      "           Sitting       0.36      0.45      0.40        22\n",
      "          Standing       0.45      0.56      0.50        27\n",
      "            Laying       1.00      0.57      0.73        28\n",
      "\n",
      "       avg / total       0.70      0.63      0.64       148\n",
      "\n"
     ]
    }
   ],
   "source": [
    "k = 1\n",
    "y_pred = find_k_closest(dist_m, y_train, k=k)\n",
    "print(\"Parameters:\")\n",
    "print(\"k = {}\".format(k))\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"Test set report\")\n",
    "print(classification_report(y_test, y_pred, target_names=label_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing we can note is that the results are better than DTW (precision: 0.64, recall: 0.60, f1-score: 0.60) and this method is faster (and more scalable) than [knn with DTW](https://github.com/jeandeducla/ML-Time-Series/blob/master/k_Nearest_Neighbors-Accelerometer-Raw.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if instead of keeping k=1 we try to optimize this parameter to achieve the best f1 score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_k_best(dist_m, y_train, y_test, k_range=np.arange(1,22)):\n",
    "    k_range = np.arange(1,22) # range of k to test\n",
    "    f1_scores = np.empty(k_range.shape) # we are going to store f1 scores here\n",
    "    # now we loop over k_range and compute f1_scores...\n",
    "    for k in k_range:\n",
    "        y_pred = find_k_closest(dist_m, y_train, k=k)\n",
    "        f1_scores[k-1] = f1_score(y_test, y_pred, average='macro')\n",
    "    return k_range[np.argmax(f1_scores)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters:\n",
      "k = 9\n",
      "\n",
      "\n",
      "Test set report\n",
      "                    precision    recall  f1-score   support\n",
      "\n",
      "           Walking       0.62      1.00      0.77        23\n",
      "  Walking upstairs       0.85      0.68      0.76        25\n",
      "Walking downstairs       1.00      0.61      0.76        23\n",
      "           Sitting       0.33      0.36      0.35        22\n",
      "          Standing       0.56      0.74      0.63        27\n",
      "            Laying       0.82      0.50      0.62        28\n",
      "\n",
      "       avg / total       0.70      0.65      0.65       148\n",
      "\n"
     ]
    }
   ],
   "source": [
    "k_best = find_k_best(dist_m, y_train, y_test, k_range=np.arange(1,22))\n",
    "y_pred = find_k_closest(dist_m, y_train, k=k_best)\n",
    "\n",
    "print(\"Parameters:\")\n",
    "print(\"k = {}\".format(k_best))\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"Test set report\")\n",
    "print(classification_report(y_test, y_pred, target_names=label_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of using features instead of raw signals is to be able to concatenate information about the three axis of the accelerometer data while this does not make sense when using knn with DTW. But let's try to use knn with feature vectors but using only features of x axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## kNN with euclidian distance and x axis feature vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are first going to re write the feature extraction function to extract features from x axis only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def feature_extraction_1_axis(x, Te=1.0):\n",
    "    \n",
    "    # Raw signals :  stat and area features\n",
    "    features_xt = stat_area_features(x, Te=Te)\n",
    "    # Jerk signals :  stat and area features\n",
    "    features_xt_jerk = stat_area_features((x[:,1:]-x[:,:-1])/Te, Te=Te)\n",
    "    # Raw signals : frequency domain features \n",
    "    features_xf = frequency_domain_features(x, Te=1/Te)    \n",
    "    # Jerk signals : frequency domain features \n",
    "    features_xf_jerk = frequency_domain_features((x[:,1:]-x[:,:-1])/Te, Te=1/Te)\n",
    "        \n",
    "    return np.concatenate((features_xt,features_xt_jerk, features_xf,features_xf_jerk), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We extract features for x axis only..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape : (368, 72)\n",
      "X_test shape: (148, 72)\n"
     ]
    }
   ],
   "source": [
    "X_train_1 = feature_extraction_1_axis(X_train_x_raw, Te=1/50)\n",
    "X_test_1 = feature_extraction_1_axis(X_test_x_raw, Te=1/50)\n",
    "\n",
    "print(\"X_train shape : {}\".format(X_train_1.shape))\n",
    "print(\"X_test shape: {}\".format(X_test_1.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler = preprocessing.StandardScaler().fit(X_train_1)\n",
    "X_train_1 = scaler.transform(X_train_1) \n",
    "X_test_1 = scaler.transform(X_test_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We compute the distance matrix..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Execution time: 0.00 min 0.56 s \n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "dist_m_1 = make_distance_matrix(X_train_1, X_test_1)\n",
    "stop = time.time()\n",
    "\n",
    "print(\"Execution time: {:.2f} min {:.2f} s \".format((stop-start) // 60, (stop-start) % 60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we find the 1 nearest neighbor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters:\n",
      "k = 1\n",
      "\n",
      "\n",
      "Test set report\n",
      "                    precision    recall  f1-score   support\n",
      "\n",
      "           Walking       0.70      0.83      0.76        23\n",
      "  Walking upstairs       0.55      0.68      0.61        25\n",
      "Walking downstairs       0.77      0.43      0.56        23\n",
      "           Sitting       0.35      0.32      0.33        22\n",
      "          Standing       0.47      0.59      0.52        27\n",
      "            Laying       0.65      0.54      0.59        28\n",
      "\n",
      "       avg / total       0.58      0.57      0.56       148\n",
      "\n"
     ]
    }
   ],
   "source": [
    "k = 1\n",
    "y_pred = find_k_closest(dist_m_1, y_train, k=k)\n",
    "print(\"Parameters:\")\n",
    "print(\"k = {}\".format(k))\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"Test set report\")\n",
    "print(classification_report(y_test, y_pred, target_names=label_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can note that the results are similar to the results we have with raw signals of x axis and [knn with DTW](https://github.com/jeandeducla/ML-Time-Series/blob/master/k_Nearest_Neighbors-Accelerometer-Raw.ipynb) (precision: 0.64, recall: 0.60, f1-score: 0.60) but the execution time here is very small compared to DTW..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now like we did before, let's try to optimize k..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters:\n",
      "k = 11\n",
      "\n",
      "\n",
      "Test set report\n",
      "                    precision    recall  f1-score   support\n",
      "\n",
      "           Walking       0.54      0.96      0.69        23\n",
      "  Walking upstairs       0.55      0.44      0.49        25\n",
      "Walking downstairs       0.90      0.39      0.55        23\n",
      "           Sitting       0.42      0.59      0.49        22\n",
      "          Standing       0.63      0.70      0.67        27\n",
      "            Laying       0.75      0.43      0.55        28\n",
      "\n",
      "       avg / total       0.64      0.58      0.57       148\n",
      "\n"
     ]
    }
   ],
   "source": [
    "k_best = find_k_best(dist_m_1, y_train, y_test, k_range=np.arange(1,22))\n",
    "y_pred = find_k_closest(dist_m_1, y_train, k=k_best)\n",
    "\n",
    "print(\"Parameters:\")\n",
    "print(\"k = {}\".format(k_best))\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"Test set report\")\n",
    "print(classification_report(y_test, y_pred, target_names=label_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}