01_Lab_1_Visualisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Mining and Exploration [INFR11007]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python environment set-up\n",
    "Part of the fist lab will be spent in setting-up a python working environvment. Follow the [instructions](https://github.com/sinziana91/DME/blob/master/environment-set-up.md) provided  and make sure your environment is set-up correctly. In case you run into trouble, please seek advice in the lab sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 1: Data exploratory analysis and visualisations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we introduce the [landsat satellite](https://archive.ics.uci.edu/ml/datasets/Statlog+%28Landsat+Satellite%29) dataset that we will be using throughout the course and  perform some exloratory data analysis and  visualisations on the dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Landsat satellite dataset\n",
    "#### Description\n",
    "The database consists of the multi-spectral values of pixels in 3x3 neighbourhoods in a satellite image, and the classification associated with the central pixel in each neighbourhood. The aim is to predict this classification, given the multi-spectral values. One frame of Landsat MSS imagery consists of four digital images of the same scene in different spectral bands. Two of  these are in the visible region (corresponding approximately to green and red regions of the visible spectrum) and two are in the (near) infra-red. Each pixel is an 8-bit binary word, with 0 corresponding to black and 255 to white. The spatial resolution of a pixel is about 80m x 80m. Each image contains 2340 x 3380 such pixels.\n",
    "\n",
    "Our database is a (tiny) sub-area of a scene, consisting of 82 x 100 pixels. Each line of data corresponds to a 3x3 square neighbourhood of pixels completely contained within the 82x100 sub-area. Each line contains the pixel values in the four spectral bands (converted to ASCII) of each of the 9 pixels in the 3x3 neighbourhood and a number indicating the classification label of the central pixel. \n",
    "\n",
    "#### Classes\n",
    "The number is a code for the following classes:\n",
    "\n",
    "1. red soil\n",
    "2. cotton crop\n",
    "3. grey soil\n",
    "4. damp grey soil\n",
    "5. soil with vegetation stubble\n",
    "6. mixture class (all types present)\n",
    "7. very damp grey soil\n",
    "\n",
    "*NB. There are no examples with class 6 in this dataset, hence there are actually 6 classes, i.e. 1,2,3,4,5 and 7.*\n",
    "\n",
    "####  Data pre-processing\n",
    "The data is given in random order and certain lines of data have been removed so that the original image cannot be reconstructed from this dataset.\n",
    "\n",
    "In each line of data, the four spectral values for the top-left pixel are given first followed by the four spectral values for the top-middle pixel and then those for the top-right pixel, and so on with the pixels read out in sequence left-to-right and top-to-bottom. \n",
    "\n",
    "#### Number of examples\n",
    "* training set:     4435\n",
    "* test set:         2000\n",
    "\n",
    "#### Number of attributes\n",
    "36 (= 4 spectral bands x 9 pixels in neighbourhood )\n",
    "\n",
    "#### Attributes\n",
    "The attributes are numerical, in the range 0 to 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the cell below to import the packages we will be using throughout this lab. The last line enforces the matplotlib figures to be rendered within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required packages \n",
    "from __future__ import division, print_function\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas\n",
    "[Pandas](http://pandas.pydata.org) is a library for data manipulation and analysis. There are two fundamental data structures in pandas: the **Series** and **DataFrame** structures which are built on top of NumPy arrays. DataFrame structures are very similar to Data Frames used in the  R programming language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by loading the  training and test datasets  into pandas DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_path = os.path.join(os.getcwd(), 'datasets', 'landsat', 'landsat_train.csv')\n",
    "test_path = os.path.join(os.getcwd(), 'datasets', 'landsat', 'landsat_test.csv')\n",
    "landsat_train = pd.read_csv(train_path, delimiter = ',')\n",
    "landsat_test = pd.read_csv(test_path, delimiter = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preliminary inspections\n",
    "We can use the `pandas` [`head()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html) method to inspect the  first  `n` entries in the DataFrame. Note that the last column in the DataFrame contains the label for each data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show first 5 instances of train set\n",
    "landsat_train.head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can use the `pandas` [`sample()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html) method to inspect  `n` random entries in the DataFrame. We can set the `random_state` parameter to ensure reproducible results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1 ==========\n",
    "Inspect 7 random entries in the test dataset. Set the `random_state` parameter to a number of your choice (i.e. `10`) to ensure reproducible results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2 ==========\n",
    "\n",
    "The `pandas` [`info()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.info.html) method provides basic information (i.e. number of rows/columns, variable data types) about a DataFrame.\n",
    "\n",
    "Display the basic information about the `landsat_train` dataframe. How many attributes/samples are there in the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "There are  4435 entries and 37 columns in the  `landsat_train` dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can use the `shape` attribute of a DataFrame to get the number of entries (i.e. rows/samples) and columns (i.e. attributes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"There are {} entries and {} columns in the landsat_train DataFrame\"\\\n",
    "      .format(landsat_train.shape[0], landsat_train.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another very useful `pandas` method is [`describe()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html) which  generates summary statistics about the columns in a DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summary statistics in train set\n",
    "landsat_train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some times we might want to compute further statistics about the columns in a DataFrame. Since `pandas` structures are built on top of `numpy` arrays, each DataFrame is essentially a two-dimensional `numpy` array. Hence, we can use `numpy` or `scipy` functions to perform any sort of transformations or compute statistics. \n",
    "The next cell computes high-order statistics (i.e. [skewness](https://en.wikipedia.org/wiki/Skewness) and [kurtosis](https://en.wikipedia.org/wiki/Skewness)) by using functions imported from [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import skew, kurtosis\n",
    "print('Skewness:\\n{}'.format(skew(landsat_train)))\n",
    "print('Kurtosis:\\n{}'.format(kurtosis(landsat_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class labels\n",
    "The class label names are stored in a separate file.  Let's load them in another DataFrame  called `landsat_labels` and inspect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read classes and display\n",
    "labels_path = os.path.join(os.getcwd(), 'datasets', 'landsat', 'landsat_classes.csv')\n",
    "landsat_labels = pd.read_csv(labels_path, delimiter = ',', index_col=0)\n",
    "landsat_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Pandas` provides the  [`to_dict()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_dict.html) method which can be used to transform  a DataFrame into a python dictionary. It will normally return a dictionary of dictionaries, one for each column in the DataFrame. Since we only have one column  in `landsat_labels`, we can use its name to access it and end up with a single dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn the labels dataframe into a dictionary\n",
    "# We only have one column in the DataFrame\n",
    "landsat_labels_dict = landsat_labels.to_dict()[\"Class\"]\n",
    "landsat_labels_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Seaborn](https://seaborn.github.io/index.html) is a visualisation library built on top of matplotlib which offers some aesthetic enhancement and, more importantly, provides some high-level functions for  \"exploring and understanding data\". Seaborn is also tightly integrated with pandas and provides support for both numpy and pandas data structures.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualising distributions\n",
    "As a first visualising step, we want to get a feel for the distribution of the various features in the dataset. \n",
    "\n",
    "For this purpose, we can use the `seaborn` [`distplot()`](http://seaborn.pydata.org/generated/seaborn.distplot.html) function which combines a histogram with a kernel density estimate plot. Make sure  you read the [documentation](http://seaborn.pydata.org/generated/seaborn.distplot.html) of this function  to understand how it can be used.\n",
    "\n",
    "We have a total of 36 features (9 pixels  $\\times$ 4 spectral bands) and we will produce one `distplot` for each feature. \n",
    "\n",
    "Execute the cell below to visualise the univariate distributions and kernel density estimates. \n",
    "\n",
    "You should spend some time here to make sure you understand what each line of code does in this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(9,4, figsize=(17,17)) # Figure with 9 rows and 4 columns\n",
    "pixels = np.arange(1,10) # Pixel values (1-9)\n",
    "bands = np.arange(1,5) # Spectral band values (1-4)\n",
    "for ii, pixel in enumerate(pixels):\n",
    "    for jj, band in enumerate(bands):\n",
    "        variable_name = 'pixel_' + str(pixel) + '_' + str(band) # Get the variable name of interest\n",
    "        sns.distplot(landsat_train[variable_name], ax=ax[ii][jj], kde=True) # Use a single feature at a time\n",
    "        ax[ii][jj].xaxis.label.set_visible(False)\n",
    "[ax[0][ii].set_title(\"Band {}\".format(band)) for ii, band in enumerate(bands)] # Set band titles for top plots\n",
    "[ax[ii][0].set_ylabel(\"Pixel {}\".format(pixel)) for ii, pixel in enumerate(pixels)] # Set pixel titles for left-most plots\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3 ==========\n",
    "It seems like intensity distributions for the different pixels are similar within the same spectral band. Is this surprising? If not, explain why."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4 ========== \n",
    "\n",
    "Given the observation made above, we now want to visualise the pixel intensity distributions by pooling all pixels together for each entry in the dataset. We still want to do this separately for each spectral band.\n",
    "\n",
    "Modify the code provided above to produce a figure with 4 subplots (one for each spectral band), and within each subplot show the distribution and kernel density estimate for the pooled pixel intensity values. For each `distplot` set the number of bins equal to 25.\n",
    "\n",
    "*Hint: the `distplot()` function accepts one-dimensional arrays. To be able to use it, you will need to transform the data. For this purpose, you might find  the [`reshape()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html) numpy function useful.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, suppose we want to visualise the pooled pixel distributions separately for every spectral band, as well as for every class in the dataset. We can do this by filtering the data according to their corresponding label, one class at a time.\n",
    "\n",
    "You are provided with sample code to achieve this. Once again, make sure you understand what every line of code does in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show distributions separately for each class and spectral band\n",
    "labels = np.sort(landsat_train.label.unique()) # Get the labels in the dataset, by looking at possible values of \"label\" attribute\n",
    "fig, ax = plt.subplots(labels.size,4, figsize=(17,14))\n",
    "for ii, label in enumerate(labels):\n",
    "    for jj, band in enumerate(bands):\n",
    "        variable_names = ['pixel_' + str(pixel) + '_' + str(band) for pixel in pixels] # Pool pixels together\n",
    "        sns.distplot(landsat_train[landsat_train[\"label\"]==label][variable_names].values.reshape(-1,), \\\n",
    "                     ax=ax[ii][jj], kde=True, bins=25) # Filter by label\n",
    "[ax[0][ii].set_title(\"Band {}\".format(band)) for ii, band in enumerate(bands)] # Set band titles on top plots\n",
    "[ax[ii][0].set_ylabel(\"{}\".format(landsat_labels_dict[label])) for ii, label in enumerate(labels)] # Set label titles in left-most plots\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the different classes can be discriminated by looking at the distribution of the pooled pixel intensities. This is good news, as it means that a classifier would hopefully be able to predict the right labels from pixel intensity values. \n",
    "\n",
    "In some cases we might want to plot the kernel density estimates for each class on top of one another to be able to *roughly* tell whether the classes are distinguishable. If we use the `distplot()` function as we have done so far, things might get a bit too messy. Instead, we can use `seaborn's` `kdeplot()` function which only plots the kernel density estimate of a variable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 5 ========== \n",
    "Produce a figure with four subplots (one for each spectral band) and within each subplot use the [`kdeplot()`](http://seaborn.pydata.org/generated/seaborn.kdeplot.html) function to show the kernel density estimate for the pooled pixel intensity values, separately for each class. Pay special attention in setting the legend(s) in your figure correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 6 ========== \n",
    "\n",
    "By observing the above kernel density estimate plots, which classes do you think are easy/difficult to separate when using pixel intensity values only?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have focused on univariate feature distributions. Now, we want to get a feel for the correlations between different features. `Seaborn ` offers the `pairplot()` function, which is an excellent tool for visualising pair-wise relationships between variables.  \n",
    "\n",
    "The following example  shows the  pairwise relationship between the features `pixel_1_1` and `pixel_1_2`. Refer to the [`pairplot`](http://seaborn.pydata.org/generated/seaborn.pairplot.html) documentation  to understand how this function can be used.  Feel free to experiment with other pairs of variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(landsat_train, vars = [\"pixel_1_1\", \"pixel_1_2\"], \\\n",
    "             plot_kws={'s' : 6}, diag_kws={'bins' : 25}) # Set variables of interest, marker size and bins for histograms\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above plot shows the pair-wise relationship between only two variables. Our feature space is 36-dimensional, so if we wanted to repeat the same procedure for each possible pair of variables  we would end up with a 36 $\\times$ 36 figure which would  not be very meaningful (also it would be fairly computationally expensive to produce). \n",
    "\n",
    "Instead, we can pool  pixels together again, as we did in the previous part. This time, instead of treating each pixel in the same way and combining all pixel values, we can compute the average pixel value in each spectral band.\n",
    "\n",
    "The following bit of code  computes the average pixel value within each spectral band, separately  for  each observation, and saves the result in a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for band in bands:\n",
    "    variable_names = ['pixel_' + str(pixel) + '_' + str(band) for pixel in pixels]\n",
    "    landsat_train['avg_' + str(band)] = landsat_train[variable_names].mean(axis=1)\n",
    "landsat_train.head(5) # Show the first 5 observations in the updated dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 7 ========== \n",
    "By using the  `seaborn` `pairplot()` function, show the pairwise correlations between the mean pixel values in each spectral band for the training set (`landsat_train`). \n",
    "\n",
    "*Hint: make appropriate use of the `vars` argument of the function.*\n",
    "\n",
    "Which spectral band pairs exhibit the strongest correlations?  Are these correlations expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 8 ========== \n",
    "The `pairplot` function can also  be used to visualise pair-wise relationships between variables, conditioned on the label, that is, separately for each class.\n",
    "\n",
    "Modify your code from the previous question to visualise pair-wise relationships between spectral bands, separately for each class. For the diagonal plots, show kernel density estimates instead of histograms  which are shown by default. Do not worry about changing the legend entries at this point.\n",
    "\n",
    "*Hint: make appropriate use of the `hue` and `diag_kind` parameters of the [`pairplot`](http://seaborn.pydata.org/generated/seaborn.pairplot.html) function.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 9 ========== \n",
    "Do you think that feature interactions can help in discrimanting the different classes? Would it make sense to use a classifier that makes use of such correlations to predict labels? For instance, would you expect a [Quadratic Discriminant Analysis (QDA)](https://en.wikipedia.org/wiki/Quadratic_classifier) classifier to perform better or worse than a [Gaussian Naive Bayes (GNB)](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Gaussian_naive_Bayes) model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}