facebookAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center'> Data Analysis on <i>\"Facebook Comment Volume\"</i> Dataset </h1>\n",
    "<h3 align='center'> Course Data Analytics and Data Driven Decisions </h3>\n",
    "\n",
    "<h5 align='center'> Davide <i>Mariotti<i> - *255558*, Luca <i>Grillo<i> - *254377*, Stefano <i>Cortellessa<i> -*254260* </h5>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction: Why this data\n",
    "\n",
    "The aim of this project is to do a deep analysis on a dataset in order to achieve a scope. In our case we have chosen the \"Facebook Comment Volume\" dataset, a set containing a lot of useful information regarding a Facebook post (like number comments, number of shares, etc.).\n",
    "The aim of our analysis is to answer at the following question: May the length of a post influence its reading and consequently also the number of comments and shares it will receive?\n",
    "For answering to this problem, we walked through two different preliminary phases:\n",
    "-\tData Cleaning of the dataset,\n",
    "-\tExploratory analysis.\n",
    "After these we performed two others Machine Learning technics named Supervised Learning and Unsupervised Learning to predict which will be the shares for a post in the future.\n",
    "\n",
    "First of all, we considered a significant number of datasets, each of which very interesting as in terms of topics covered as for their attribute’s configuration. We gradually discarded some of them, keeping all sets that at the same time had a better structure and aroused our interest.\n",
    "In the end we opted for \"Facebook Comment Volume\", a dataset containing a lot of useful information regarding a Facebook post. \n",
    "The most important reason we have chosen the Facebook dataset was the argument factor; we were looking for a set in which we could apply all concepts learned during the course, but at the same time the dataset had to capture our attention and it had to be stimulant for subsequent studies and analysis. A second criterion was dimension; we were aiming for a sufficiently large dataset, both regarding entries and attributes. In fact, working with a large number of information, we are able to produce more consistent results for a large number of studies, like explorative analysis, supervised learning and unsupervised learning. A third parameter for the choice was the structure; the dataset on which we desired to apply our work had to contain significant values, namely attributes able to adequately describe the application reality of interest. \n",
    "\n",
    "Some important domain specific concepts are discussed below:\n",
    "- Public Group/Facebook Page: It is a public profile specifically created for businesses, brands, celebrities etc.\n",
    "- Post/Feed: These are basically the individual stories published on page by administrators of page.\n",
    "- Comment: It is an important activity in social sites, that gives potential to become a discussion forum and it is only one measure of popularity/interest towards post is to which extent readers are inspired to leave comments on document/post.\n",
    "- Share: It is another important activity in social sites, that allow people to share posts so that other people can read and comment on these posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Description\n",
    "\n",
    "In the following table the meaning of each column is explained:\n",
    "\n",
    "<table width=\"770\" cellspacing=\"0\" cellpadding=\"7\">\n",
    "<tbody>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\" height=\"20\">\n",
    "<p align=\"center\"><span style=\"font-size: medium;\"><span lang=\"en-GB\"><strong>Column Name</strong></span></span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p align=\"center\"><span style=\"font-size: medium;\"><span lang=\"en-GB\"><strong>Column Description</strong></span></span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Page Popularity/likes</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">It is a feature that defines users support for specific comments, pictures, wall posts, statuses, or pages.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Page Checkin's</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Describes how many individuals so far visited this place. This feature is only associated with the places eg:some institution, place, theater etc.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Page Talking About</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">This is the actual count of users who are ’engaged’ and interacting with that Facebook Page. The users who actually come back to the page, after liking the page. This include activities such as comments, likes to a post, shares by visitors to the page.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Page Category</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">This defined the category of source of document eg: Local business or place, brand or product, company or institution, artist, band, entertainment, community etc. The category is defined by an intenger number. </span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">C1</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Total comment count before selected base date/time.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Comment count in last 24 hrs w.r.t to selected base date/time.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Comment count is last 48 hrs to last 24 hrs w.r.t to base date/time.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Comment count in first 24 hrs after publishing the document, but before the selected base date/time.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">The difference between C2 and C3.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">min C1</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the min of the variable C1 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">max C1</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the max of the variable C1 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">mean C1</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Mean of the variable C1 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">median C1</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Median of the variable C1 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">standard_deviation C1 </span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Standard deviation of the variable C1 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">min C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the min of the variable C2 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">max C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Max of the variable C2 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">mean C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Mean of the variable C2 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">median C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Median of the variable C2 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">standard_deviation C2</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Standard deviation of the variable C2 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">min C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the min of the variable C3 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">max C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the max of the variable C3 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">mean C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Mean of the variable C3 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">median C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the median of the variable C3 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">standard_deviation C3</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Standard deviation of the variable C3 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">min C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the min of the variable C4 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">max C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the max of the variable C4 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">mean C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Mean of the variable C4 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">median C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Median of the variable C4 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">standard_deviation C4</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Standard deviation of the variable C4 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">min C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the min of the variable C5 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">max C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Max of the variable C5 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">mean C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the Mean of the variable C5 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">median C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the median of the variable C5 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">standard_deviation C5</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Define the standard deviation of the variable C5 grouped by pages</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Base Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Selected time in order to simulate the scenario. Decimal(0-71) Encoding</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Post length</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Character count in the post.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Post Share Count</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">This features counts the number of shares of the post, that how many peoples had shared this post on to their timeline.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Post Promotion Status</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">To reach more people with posts in News Feed, individual promote their post and this features tells that whether the post is promoted(1) or not(0).</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">H Local</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">This describes the H hrs, for which we have the target variable/ comments received.</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Monday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Monday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Tuesday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Tuesday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Wednsday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Wednesday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Thursday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Thursday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Friday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Friday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Saturday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Saturday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Sunday</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates if the post was posted on Sunday (0-1)</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Monday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    " <tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Tuesday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Wednesday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Thursday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Friday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Saturday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Sunday_Base_Time</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">Indicates the day on which the post was published on selected base date/time</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"162\">\n",
    "<p><span lang=\"en-GB\">Target Variable</span></p>\n",
    "</td>\n",
    "<td style=\"border: 1px solid #00000a; padding: 0cm 0.19cm 0cm 0.2cm;\" width=\"451\">\n",
    "<p><span lang=\"en-GB\">The number of comments in next H hrs (H is the variable 'H Local').</span></p>\n",
    "</td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "The dataset can be found at this link:\n",
    "http://archive.ics.uci.edu/ml/datasets/Facebook+Comment+Volume+Dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning\n",
    "\n",
    "In this section we are going to clean the data before we start our analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all libraries needed \n",
    "import pandas as pd\n",
    "import math\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "df = pd.read_csv('Features_Variant_1.csv', sep=',',  header=None)\n",
    "\n",
    "# Display the dataset\n",
    "pd.set_option('display.max_columns', 100)\n",
    "pd.set_option('display.max_rows', 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Add headers (columns) to the dataset according to the dataset specification\n",
    "df = df.rename(index=str, columns={0:\"Likes\", 1:\"Page Checkins\", 2:\"Page talking about\", 3:\"Page Category\", 4:\"min C1\", 5:\"max C1\", 6:\"mean C1\", 7:\"median C1\", 8:\"standard_deviation C1\", 9:\"min C2\", 10:\"max C2\", 11:\"mean C2\", 12:\"median C2\", 13:\"standard_deviation C2\", 14:\"min C3\", 15:\"max C3\", 16:\"mean C3\", 17:\"median C3\", 18:\"standard_deviation C3\", 19:\"min C4\", 20:\"max C4\", 21:\"mean C4\", 22:\"median C4\", 23:\"standard_deviation C4\", 24:\"min C5\", 25:\"max C5\", 26:\"mean C5\", 27:\"median C5\", 28:\"standard_deviation C5\", 29:\"C1\", 30:\"C2\", 31:\"C3\", 32:\"C4\", 33:\"C5\", 34:\"Base_Time\", 35:\"Post length\", 36:\"Post Share Count\", 37:\"Post Promotion Status\", 38:\"H Local\", 39:\"Monday\", 40:\"Tuesday\", 41:\"Wednesday\", 42:\"Thursday\", 43:\"Friday\", 44:\"Saturday\", 45:\"Sunday\", 46:\"Monday_Base_Time\", 47:\"Tuesday_Base_Time\", 48:\"Wednesday_Base_Time\", 49:\"Thursday_Base_Time\", 50:\"Friday_Base_Time\", 51:\"Saturday_Base_Time\", 52:\"Sunday_Base_Time\", 53:\"Target Variable\"})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check columns with NaN values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.isna().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no columns with NaN values, so we can leave the dataframe as it is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to be sure that data in our dataset are correct. So: \n",
    "- We check if all the values in Page Checkin's, Likes, Page talking about, C1, C2, C3, C4, Base Time, Post length and Post share count are positive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Page Checkins control\n",
    "if len(df[df['Page Checkins'] < 0]['Page Checkins']) > 0:\n",
    "    print('Page Checkins contains non-positive values')\n",
    "else:\n",
    "    print('All values in Page Checkins column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Likes control\n",
    "if len(df[df['Likes'] < 0]['Likes']) > 0:\n",
    "    print('Likes contains non-positive values')\n",
    "else:\n",
    "    print('All values in Likes column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Page talking about control\n",
    "if len(df[df['Page talking about'] < 0]['Page talking about']) > 0:\n",
    "    print('Page talking about contains non-positive values')\n",
    "else:\n",
    "    print('All values in Page talking about column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# C1 control\n",
    "if len(df[df['C1'] < 0]['C1']) > 0:\n",
    "    print('C1 contains non-positive values')\n",
    "else:\n",
    "    print('All values in C1 column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# C2 control\n",
    "if len(df[df['C2'] < 0]['C2']) > 0:\n",
    "    print('C2 contains non-positive values')\n",
    "else:\n",
    "    print('All values in C2 column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# C3 control\n",
    "if len(df[df['C3'] < 0]['C3']) > 0:\n",
    "    print('C3 contains non-positive values')\n",
    "else:\n",
    "    print('All values in C3 column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# C4 control\n",
    "if len(df[df['C4'] < 0]['C4']) > 0:\n",
    "    print('C4 contains non-positive values')\n",
    "else:\n",
    "    print('All values in C4 column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Base Time control\n",
    "if len(df[df['Base_Time'] < 0]['Base_Time']) > 0:\n",
    "    print('Base Time contains non-positive values')\n",
    "else:\n",
    "    print('All values in Base Time column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Post length control\n",
    "if len(df[df['Post length'] < 0]['Post length']) > 0:\n",
    "    print('Post length contains non-positive values')\n",
    "else:\n",
    "    print('All values in Post length column are positive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Post Share Count control\n",
    "if len(df[df['Post Share Count'] < 0]['Post Share Count']) > 0:\n",
    "    print('Post Share Count contains non-positive values')\n",
    "else:\n",
    "    print('All values in Post Share Count column are positive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check if all the values in the days columns are 0 or 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_monday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Monday'] != 0:\n",
    "            if row['Monday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Monday values are all valid\")\n",
    "    else:\n",
    "        print(\"Monday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_tuesday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Tuesday'] != 0:\n",
    "            if row['Tuesday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Tuesday values are all valid\")\n",
    "    else:\n",
    "        print(\"Tuesday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_wednesday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Wednesday'] != 0:\n",
    "            if row['Wednesday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Wednesday values are all valid\")\n",
    "    else:\n",
    "        print(\"Wednesday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_thursday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Thursday'] != 0:\n",
    "            if row['Thursday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Thursday values are all valid\")\n",
    "    else:\n",
    "        print(\"Thursday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_friday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Friday'] != 0:\n",
    "            if row['Friday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Friday values are all valid\")\n",
    "    else:\n",
    "        print(\"Friday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_saturday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Saturday'] != 0:\n",
    "            if row['Saturday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Saturday values are all valid\")\n",
    "    else:\n",
    "        print(\"Saturday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_sunday(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Sunday'] != 0:\n",
    "            if row['Sunday'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Sunday values are all valid\")\n",
    "    else:\n",
    "        print(\"Sunday values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_monday(df)\n",
    "check_tuesday(df)\n",
    "check_wednesday(df)\n",
    "check_thursday(df)\n",
    "check_friday(df)\n",
    "check_saturday(df)\n",
    "check_sunday(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check that all the values in Base Data Time are valid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_monday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Monday_Base_Time'] != 0:\n",
    "            if row['Monday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Monday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Monday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_tuesday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Tuesday_Base_Time'] != 0:\n",
    "            if row['Tuesday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Tuesday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Tuesday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_wednesday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Wednesday_Base_Time'] != 0:\n",
    "            if row['Wednesday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Wednesday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Wednesday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_thursday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Thursday_Base_Time'] != 0:\n",
    "            if row['Thursday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Thursday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Thursday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_friday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Friday_Base_Time'] != 0:\n",
    "            if row['Friday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Friday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Friday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_saturday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Saturday_Base_Time'] != 0:\n",
    "            if row['Saturday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Saturday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Saturday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_sunday_base_time(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['Sunday_Base_Time'] != 0:\n",
    "            if row['Sunday_Base_Time'] != 1:\n",
    "                rows.append(index)\n",
    "                check = False\n",
    "        \n",
    "    if check:\n",
    "        print(\"Sunday_Base_Time values are all valid\")\n",
    "    else:\n",
    "        print(\"Sunday_Base_Time values are not all valid, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_monday_base_time(df)\n",
    "check_tuesday_base_time(df)\n",
    "check_wednesday_base_time(df)\n",
    "check_thursday_base_time(df)\n",
    "check_friday_base_time(df)\n",
    "check_saturday_base_time(df)\n",
    "check_sunday_base_time(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check if the values inside C1 are correct. In particular, we are going to check if:\n",
    "    - C1 >= C2 + C3\n",
    "    - C4 <= C1 \n",
    "    - C5 = C2 - C3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_comment1(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['C1'] < row['C2'] + row['C3']:\n",
    "            rows.append(index)\n",
    "            check = False\n",
    "\n",
    "    if check:\n",
    "        print(\"All C1 variables are GRATER than C2 + C3\")\n",
    "    else:\n",
    "        print(\"Not all C1 variables are LESS than C2 + C3, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_comment2(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['C4'] > row['C1']:\n",
    "            rows.append(index)\n",
    "            check = False\n",
    "\n",
    "    if check:\n",
    "        print(\"All C1 variables are GRATER than C4\")\n",
    "    else:\n",
    "        print(\"Not all C1 variables are LESS than C4, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_comment3(df, check=True, rows=[]):\n",
    "    for index, row in df.iterrows():\n",
    "        if row['C5'] != row['C2'] - row['C3']:\n",
    "            rows.append(index)\n",
    "            check = False\n",
    "\n",
    "    if check:\n",
    "        print(\"All C5 variables are EQUAL to C2 - C3\")\n",
    "    else:\n",
    "        print(\"Not all C5 variables are NOT EQUAL to C2 - C3, specifically the following rows:\")\n",
    "        for i in range(len(rows)):\n",
    "            print (rows[i])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_comment1(df)\n",
    "check_comment2(df)\n",
    "check_comment3(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### After this checking phase we can say that all the values are correct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useless Columns Removal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The columns Post Promotion Status and H Local are not very significant for our analysis, so we are going to check if we may remove them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = True\n",
    "for index, row in df.iterrows():\n",
    "    if row['Post Promotion Status'] != 0:\n",
    "        result = False\n",
    "        \n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, row in df.iterrows():\n",
    "    if row['H Local'] != 24:\n",
    "        result = False\n",
    "        \n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can remove the Post Promotion Status column because the values are all 0. <br>\n",
    "Instead, we are not going to remove the H Local column because the values are not all 24 and it may be important for our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop('Post Promotion Status', 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to look for duplicates in the dataset. In particular, we consider two rows as duplicates if they have the same values on the all columns. We drop all rows representing duplicates. The output of the next cell indicates the number of duplicates found in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"is_duplicated\"] = df.duplicated()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Printing of duplicates\n",
    "df.loc[df['is_duplicated'] == True]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We deleted the duplicates and the column related to the duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = df.drop(df[df.is_duplicated == True].index)\n",
    "df = df.drop('is_duplicated', 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here we start our statistical analysis showing statistics about Post length variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the histogram below, the post length is concentrated between 0 and 200. We can easily see that 7000 posts have length 0. This kind of post are photos, links or videos without any description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numbers of characters for posts\n",
    "post_length_descr = df['Post length'].describe()\n",
    "print('statistics about Post length:')\n",
    "post_length_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Post length'].hist(bins=100,figsize=(20,10), range=[0, 2000])\n",
    "plt.title('Histogram of Post length')\n",
    "plt.ylabel(\"Number of Posts\")\n",
    "plt.xlabel(\"Post length\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The graphs below show the number of posts that were published every day. <br>\n",
    "    - 0 means that a post it was not published, \n",
    "    - 1 otherwise.\n",
    "\n",
    "Among the 40,000 total posts that are present in the dataset (both 0 and 1 in the graphs), those with 0 were not published that day, while every day were published n (those with 1) posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(4,2, figsize=(20,20));\n",
    "\n",
    "df['Monday'].dropna().hist(bins=100, ax=ax[0,0], range=[0,1], density=True)\n",
    "ax[0,0].set_title('Histogram of Post Published on Monday')\n",
    "                  \n",
    "df['Tuesday'].dropna().hist(bins=100, ax=ax[0,1], range=[0,1])\n",
    "ax[0,1].set_title('Histogram of Post Published on Tuesday')\n",
    "\n",
    "df['Wednesday'].dropna().hist(bins=100, ax=ax[1,0], range=[0,1])\n",
    "ax[1,0].set_title('Histogram of Post Published on Wednesday')\n",
    "\n",
    "df['Thursday'].dropna().hist(bins=100, ax=ax[1,1], range=[0,1])\n",
    "ax[1,1].set_title('Histogram of Post Published on Thursday')\n",
    "\n",
    "df['Friday'].dropna().hist(bins=100, ax=ax[2,0], range=[0,1])\n",
    "ax[2,0].set_title('Histogram of Post Published on Friday')\n",
    "\n",
    "df['Saturday'].dropna().hist(bins=100, ax=ax[2,1], range=[0,1])\n",
    "ax[2,1].set_title('Histogram of Post Published on Saturday')\n",
    "\n",
    "df['Sunday'].dropna().hist(bins=100, ax=ax[3,0], range=[0,1])\n",
    "ax[3,0].set_title('Histogram of Post Published on Sunday')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph below shows better the number of posts published each days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Comparison between the number of posts published each day\n",
    "monday = len(df[df['Monday'] == 1 ].index)\n",
    "tuesday = len(df[df['Tuesday'] == 1 ].index)\n",
    "wednesday = len(df[df['Wednesday'] == 1 ].index)\n",
    "thursday = len(df[df['Thursday'] == 1 ].index)\n",
    "friday = len(df[df['Friday'] == 1 ].index)\n",
    "saturday = len(df[df['Saturday'] == 1 ].index)\n",
    "sunday = len(df[df['Sunday'] == 1 ].index)\n",
    "plot_days = pd.Series({'Monday': monday,\n",
    "               'Tuesday' : tuesday, 'Wednesday' : wednesday, 'Thursday' : thursday,\n",
    "               'Friday' : friday, 'Saturday' : saturday, 'Sunday' : sunday,})\n",
    "\n",
    "plot = plot_days.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.title('Number of posts for each day')\n",
    "plt.ylabel('Number of posts')\n",
    "plt.xlabel('Week-Day')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Around 5000 are published every day as we can see from the histograms above. <br>\n",
    "\n",
    "It is also easy to see that **thursday** is the day with the *highest number of pubblication* with around 6000 posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Statistics about number of comments (C2, C3 and C4) of all the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Values for all pages of the dataset\n",
    "c1_descr = df['C1'].describe()\n",
    "print('statistics about C1:')\n",
    "c1_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c2_descr = df['C2'].describe()\n",
    "print('statistics about C2:')\n",
    "c2_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c3_descr = df['C3'].describe()\n",
    "print('statistics about C3:')\n",
    "c3_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c4_descr = df['C4'].describe()\n",
    "print('statistics about C4:')\n",
    "c4_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We show the comments mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['C4'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the pie chart below we can see that most of the comments are made during the period C4, so between 48 and 72 hours from the BaseDataTime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plotPie = pd.DataFrame({'': [df['C2'].mean(), df['C3'].mean(), df['C4'].mean()]},\n",
    "                           index=['C2', 'C3', 'C4'])\n",
    "plot = plotPie.plot.pie(y='', figsize=(7, 7), autopct='%1.1f%%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The histograms below show the trend of comments in the 4 ranges, respectively C1, C2, C3 and C4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2,2, figsize=(20,20));\n",
    "\n",
    "df['C1'].dropna().hist(bins=100, ax=ax[0,0], range=[0, 800])\n",
    "ax[0,0].set_title('Histogram of C1')\n",
    "ax[0,0].set_ylabel('Number of posts')\n",
    "ax[0,0].set_xlabel('Number of Comments')#check that it is comment\n",
    "                  \n",
    "df['C2'].dropna().hist(bins=100, ax=ax[0,1], range=[0, 800])\n",
    "ax[0,1].set_title('Histogram of C2')\n",
    "ax[0,1].set_ylabel('Number of posts')\n",
    "ax[0,1].set_xlabel('Number of Comments')#check that it is comment\n",
    "\n",
    "df['C3'].dropna().hist(bins=100, ax=ax[1,0], range=[0, 800])\n",
    "ax[1,0].set_title('Histogram of C3')\n",
    "ax[1,0].set_ylabel('Number of posts')\n",
    "ax[1,0].set_xlabel('Number of Comments')#check that it is comment\n",
    "\n",
    "df['C4'].dropna().hist(bins=100, ax=ax[1,1], range=[0, 800])\n",
    "ax[1,1].set_title('Histogram of C4')\n",
    "ax[1,1].set_ylabel('Number of posts')\n",
    "ax[1,1].set_xlabel('Number of Comments')#check that it is comment\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The two box plots below show the statistics about the variables ***Page Talking About*** and ***Post Share Count***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Page talking about statistics\n",
    "page_talking_about_descr = df['Page talking about'].describe()\n",
    "\n",
    "page_talking_about_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxPlot = pd.DataFrame(df['Page talking about'],\n",
    "                   columns=['Page talking about'])\n",
    "boxplot = boxPlot.boxplot(figsize=(7,7), column=['Page talking about'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Post share count statistics\n",
    "page_talking_about_descr = df['Post Share Count'].describe()\n",
    "\n",
    "page_talking_about_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxPlot = pd.DataFrame(df['Post Share Count'],\n",
    "                   columns=['Post Share Count'])\n",
    "boxplot = boxPlot.boxplot(figsize=(7,7), column=['Post Share Count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here are shown likes statistics for all pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "likes_descr = df['Likes'].describe()\n",
    "print('statistics about Likes:')\n",
    "likes_descr.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Likes'].hist(bins=100,figsize=(20,10), range=[0, 2000000])\n",
    "plt.title('Histogram of Likes')\n",
    "plt.xlabel('Likes')\n",
    "plt.ylabel('Pages')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***The previous histogram shows that around 7000 posts have a very small number of likes.*** <br>\n",
    "Thus, decreasing the range between 0 and 500, we figured out the exact number of likes. Moreover, we checked that there were no pages with 0 likes, since they would not be useful for our future research."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Likes'].hist(bins=100,figsize=(20,10), range=[0, 500])\n",
    "plt.title('Histogram of Likes')\n",
    "plt.xlabel('Likes')\n",
    "plt.ylabel('Pages')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In this section we are going to show which is the post category most recurrent. <br>\n",
    "\n",
    "Categories in the dataset are presented as integer values. Thus, we loaded the legends about this values and edit our dataset in oder to have a better readbility of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "category = pd.read_csv('Categories.csv', sep=',',  header=None)\n",
    "category = category.rename(index=str, columns={0:\"Category\"})\n",
    "\n",
    "category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Insert Category in the dataset\n",
    "category_df = category.as_matrix()\n",
    "\n",
    "df['Page Category'] = df['Page Category'].astype(int)\n",
    "df[\"Category\"] = category_df[df['Page Category'] - 1]\n",
    "df = df.drop('Page Category', 1)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Category'].hist(bins=200,figsize=(20,10), stacked=False)\n",
    "plt.title('Histogram of Category')\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.xlabel('Categories')\n",
    "plt.ylabel('Number of pages')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the graph above ***Professional Sports Team*** is the most recurrent category. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## May the length of a post influence its reading and consequently also the number of comments and shares it will receive?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the section below we will try to answer at this question dividing the posts between short and long posts. Looking at several Facebook posts, we indicate with short posts those posts with a maximum of 400 characters. As a result, long posts will be those with at least 400 characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all: \n",
    "\n",
    "- We are going to show all the posts that have been posted in the first 24h, between 24h and 48h and between 48h and 72h (These hours are in reference to the base time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "post_last_24_hours = len(df[df[\"Base_Time\"] <= 24])\n",
    "temp = df[df[\"Base_Time\"] > 24]\n",
    "post_between_24_and_48 = len(temp[temp[\"Base_Time\"] <= 48])\n",
    "post_more_than_48_hours = len(df[df[\"Base_Time\"] > 48])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_posts = pd.Series({'First 24 hours': post_last_24_hours,\n",
    "               'Between 24 hours and 48 hours' : post_between_24_and_48, 'More than 48 hours' : post_more_than_48_hours\n",
    "               })\n",
    "\n",
    "plot = plot_posts.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_pie_post = pd.DataFrame({'': [post_last_24_hours, post_between_24_and_48, post_more_than_48_hours]},\n",
    "                           index=['Last 24 hours', 'Between 24 hours and 48 hours', 'More than 48 hours'])\n",
    "plot = plot_pie_post.plot.pie(y='', figsize=(7, 7), autopct='%1.1f%%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, *we can start our research*, posts are divided in:\n",
    " - ***Long posts***, with more then 400 characters\n",
    " - ***Short posts***, with less then 400 characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_short_total = df[df[\"Post length\"] <= 400]\n",
    "df_long_total = df[df[\"Post length\"] > 400]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Number of long and short posts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of long and short posts\n",
    "plot_share = pd.Series({'Short': df_short_total['Likes'].count(),\n",
    "                         'Long' : df_long_total['Likes'].count()\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is easy to see, the ***short posts are a significantly higher number***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Number of shares about long and short posts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Number of shares about long and short posts\n",
    "plot_share = pd.Series({'Short': sum(df_short_total['Post Share Count']),\n",
    "                         'Long' : sum(df_long_total['Post Share Count'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is easy to see, the ***short posts achieve a significantly higher number of shares***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We are now going to perform the same study with all the posts published in the first 24h, between 24h and 48h and between 48h and 72h (These hours are in reference to the base time) ***separately***."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_24 = df[df[\"Base_Time\"] <= 24]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_48 = temp[(temp[\"Base_Time\"] <= 48) & (temp[\"Base_Time\"] > 24)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_greater_than_48 = df[df[\"Base_Time\"] > 48]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How many posts have been shared in the first 24h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_24_short = df_less_than_24[df_less_than_24[\"Post length\"] <= 400]\n",
    "df_less_than_24_long = df_less_than_24[df_less_than_24[\"Post length\"] > 400]\n",
    "\n",
    "df_less_than_24_short = df_less_than_24_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_less_than_24_short['Post Share Count']),\n",
    "                         'Long' : sum(df_less_than_24_long['Post Share Count'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How many posts have been shared between 24h and 48h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_48_short = df_less_than_48[df_less_than_48[\"Post length\"] <= 400]\n",
    "df_less_than_48_long = df_less_than_48[df_less_than_48[\"Post length\"] > 400]\n",
    "\n",
    "df_less_than_48_short = df_less_than_48_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_less_than_48_short['Post Share Count']),\n",
    "                         'Long' : sum(df_less_than_48_long['Post Share Count'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How many posts have been shared between 48h and 72h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_greater_than_48_short = df_greater_than_48[df_greater_than_48[\"Post length\"] <= 400]\n",
    "df_greater_than_48_long = df_greater_than_48[df_greater_than_48[\"Post length\"] > 400]\n",
    "\n",
    "df_greater_than_48_short = df_greater_than_48_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_greater_than_48_short['Post Share Count']),\n",
    "                         'Long' : sum(df_greater_than_48_long['Post Share Count'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can be easily seen is that ***short posts are those with more shares***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to perform the same study with the comments that a posts may receive in the first 24h, between 24h and 48h and between 48h and 72h (These hours are in reference to the base time)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The number of comments that the posts have been received in the first 24h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_24_short = df_less_than_24[df_less_than_24[\"Post length\"] <= 400]\n",
    "df_less_than_24_long = df_less_than_24[df_less_than_24[\"Post length\"] > 400]\n",
    "\n",
    "df_less_than_24_short = df_less_than_24_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_less_than_24_short['C2']),\n",
    "                         'Long' : sum(df_less_than_24_long['C2'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The number of comments that the posts have been received between 24h and 48h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_less_than_48_short = df_less_than_48[df_less_than_48[\"Post length\"] <= 400]\n",
    "df_less_than_48_long = df_less_than_48[df_less_than_48[\"Post length\"] > 400]\n",
    "\n",
    "df_less_than_48_short = df_less_than_48_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_less_than_48_short['C3']),\n",
    "                         'Long' : sum(df_less_than_48_long['C3'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The number of comments that the posts have been received between 48h and 72h (divided between short and long)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_greater_than_48_short = df_greater_than_48[df_greater_than_48[\"Post length\"] <= 400]\n",
    "df_greater_than_48_long = df_greater_than_48[df_greater_than_48[\"Post length\"] > 400]\n",
    "\n",
    "df_greater_than_48_short = df_greater_than_48_short.reset_index()\n",
    "\n",
    "plot_share = pd.Series({'Short': sum(df_greater_than_48_short['C4']),\n",
    "                         'Long' : sum(df_greater_than_48_long['C4'])\n",
    "                        })\n",
    "\n",
    "plot = plot_share.plot.bar(figsize=(15,8))\n",
    "\n",
    "plt.xticks(rotation=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from all the study performed above, ***the length of a post might influence its shares and the number of comments***. <br>\n",
    "So:\n",
    "\n",
    "- We figured out a possible 'correct' post length. \n",
    "\n",
    "We ordered in descending order the dataframe based on the number of shares and we took the first 100 rows. Thereafter, we ordered again in descending order the resulting rows based on the number of all the comments and we took the first 50 rows. By averaging the length of the post between these first 50 rows, *we discovered that the 'correct' post length is around **107** *."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sorted = df.loc[df['Post length'] > 0]\n",
    "df_sorted = df_sorted.sort_values( ['Post Share Count'], ascending=False)\n",
    "df_sorted = df_sorted[:100]\n",
    "df_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sorted = df_sorted.sort_values(['C1'], ascending=False)\n",
    "df_sorted = df_sorted[:50]\n",
    "df_sorted['Post length'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised learning\n",
    "\n",
    "In this section we are going to perform Unsupervised learning. Since the values in our dataset are all numerical we can use a **K-means analysis**. The only exception concerns the category column, being a categorical variable. Thus, the standard k-means algorithm is not directly applicable to categorical data, for various reasons. The sample space for categorical data is discrete, and doesn't have a natural origin. For instance, a Euclidean distance function on such a space is not really meaningful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-means analysis does not support missing values in the data. As we checked in the beginning of our work, our dataset does not have any NaN values. We can proceed with the K-Means analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subset clustering variables \n",
    "cluster = df[['Likes','Page talking about','C1', 'C2', 'C3', 'C4','Base_Time','Post length', 'Post Share Count', 'Monday', 'Tuesday',\n",
    "              'Wednesday', 'Thursday','Friday','Saturday', 'Sunday']]\n",
    "cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On cluster analysis algorithms all depend on the concept of measuring the distance (or some other measure of similarity) between the different observations that we are trying to cluster. If one of the variables is measured on a much larger scale than the other variables, then whatever measure we use will be overly influenced by that variable. For this reason we decided to standardize our variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#standarize cluster variable to have mean=0 and std=1\n",
    "clustervar = cluster.copy()\n",
    "clustervar['Likes'] = preprocessing.scale(clustervar['Likes'].astype('float64'))\n",
    "clustervar['Page talking about'] = preprocessing.scale(clustervar['Page talking about'].astype('float64'))\n",
    "clustervar['C1'] = preprocessing.scale(clustervar['C1'].astype('float64'))\n",
    "clustervar['C2'] = preprocessing.scale(clustervar['C2'].astype('float64'))\n",
    "clustervar['C3'] = preprocessing.scale(clustervar['C3'].astype('float64'))\n",
    "clustervar['C4'] = preprocessing.scale(clustervar['C4'].astype('float64'))\n",
    "clustervar['Base_Time'] = preprocessing.scale(clustervar['Base_Time'].astype('float64'))\n",
    "clustervar['Post length'] = preprocessing.scale(clustervar['Post length'].astype('float64'))\n",
    "clustervar['Post Share Count'] = preprocessing.scale(clustervar['Post Share Count'].astype('float64'))\n",
    "clustervar['Monday'] = preprocessing.scale(clustervar['Monday'].astype('float64'))\n",
    "clustervar['Tuesday'] = preprocessing.scale(clustervar['Tuesday'].astype('float64'))\n",
    "clustervar['Wednesday'] = preprocessing.scale(clustervar['Wednesday'].astype('float64'))\n",
    "clustervar['Thursday'] = preprocessing.scale(clustervar['Thursday'].astype('float64'))\n",
    "clustervar['Friday'] = preprocessing.scale(clustervar['Friday'].astype('float64'))\n",
    "clustervar['Saturday'] = preprocessing.scale(clustervar['Saturday'].astype('float64'))\n",
    "clustervar['Sunday'] = preprocessing.scale(clustervar['Sunday'].astype('float64'))\n",
    "clustervar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we split the dataset in training and test sets. The first consists of 70% of the observations and the second consists of the other 30% of the observations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#split dataset into two set \n",
    "clus_train, clus_test = train_test_split(clustervar, test_size=.3, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using k-means clustering, users need some way to determine whether they are using the right number of clusters.\n",
    "One method to validate the number of clusters is the elbow method. The idea of the elbow method is to run k-means clustering on the dataset for a range of values of k (k from 1 to 10 in our case), and for each value of k calculate the ***Sum of Squared Errors*** (SSE). Then, plot a line chart of the SSE for each value of k. If the line chart looks like an arm, then the \"elbow\" on the arm is the value of k that is the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# k-means clusters analysis for 1-9 cluster\n",
    "from scipy.spatial.distance import cdist\n",
    "clusters = range(1,10)\n",
    "meandist = []\n",
    "\n",
    "for k in clusters:\n",
    "    model = KMeans(n_clusters=k)\n",
    "    model.fit(clus_train)\n",
    "    classassign = model.predict(clus_train)\n",
    "    meandist.append(sum(np.min(cdist(clus_train, model.cluster_centers_, 'euclidean'), axis=1))\n",
    "                    / clus_train.shape[0])\n",
    "\n",
    "plt.plot(clusters, meandist)\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('Avarage distance')\n",
    "plt.title('Selecting K with the elbow method')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the average distance decreases when the number of cluster increases. We want to choose a fewer number of clusters that provide a low avarage distance. \n",
    "In the cell below we are going to interpret the ***8 cluster solution***. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Interpreting 8 cluster solution\n",
    "model8 = KMeans(n_clusters=8)\n",
    "model8.fit(clus_train)\n",
    "classassign = model8.predict(clus_train)\n",
    "\n",
    "# Plot clusters\n",
    "from sklearn.decomposition import PCA\n",
    "pca_2 = PCA(2)\n",
    "plot_columns = pca_2.fit_transform(clus_train)\n",
    "centers = pca_2.transform(model8.cluster_centers_)\n",
    "print(centers)\n",
    "\n",
    "plt.scatter(x=plot_columns[:,0],y=plot_columns[:,1],c=model8.labels_)\n",
    "plt.scatter(centers[:,0], centers[:,1], marker=\"x\", color='r')\n",
    "plt.xlabel('Canonical variable 1')\n",
    "plt.ylabel('Canonical variable 2')\n",
    "plt.title('Scatterplot of canonical variables for 8 clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scatterplot above shows that the seven clusters excluding the one more spread out are dense pack meaning that the observation within the clusters are pretty highly correlated between each other and within cluster's variance is relative low. This overlap means that there is not good separation between these seven clusters. On the other hand, the cluster with center at position around (8, -0.7) shows a better separation and the observations are more spread out indicating less correlation among the observations and high within cluster's variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable below shows the centroids of the K clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centroids = []\n",
    "centroids.append(pd.DataFrame(model8.cluster_centers_, columns=cluster.columns))\n",
    "centroids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to carry out the Supervised Learning task on the Dataset, among all possible Machine Learning algorithms we have chosen to use the Decision Tree Learning and Neural Network Learning as predictive model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we predicted the target variable 'Class' that represents a specific category of ***the number of post shares***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, the number of shares of a post has been divided in 4 different categories that indicate respectively:\n",
    "- A ***low*** number of shares that indicates a number between 1 and 49 and it is identified by the number '1'\n",
    "- A ***medium*** number of shares that indicates a number between 50 and 199 and it is identified by the number '2'\n",
    "- A ***good*** number of shares that indicates a number between 200 and 699 and it is identified by the number '3'\n",
    "- An ***high*** number of shares that indicates a number grater then 700 and it is identified by the number '4'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to create 'class' column\n",
    "def create_class(df):\n",
    "    dfNew = df.copy()\n",
    "\n",
    "    dfNew.loc[(dfNew[\"Post Share Count\"] >= 1) & (dfNew[\"Post Share Count\"] < 50), \"Class\"] = '1'\n",
    "    dfNew.loc[(dfNew[\"Post Share Count\"] >= 50) & (dfNew[\"Post Share Count\"] < 200), \"Class\"] = '2'\n",
    "    dfNew.loc[(dfNew[\"Post Share Count\"] >= 200) & (dfNew[\"Post Share Count\"] < 700), \"Class\"] = '3'\n",
    "    dfNew.loc[(dfNew[\"Post Share Count\"] >= 700)] = '4'\n",
    "    \n",
    "    return dfNew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees are a powerful prediction method and extremely popular.\n",
    "It works for both continuous as well as categorical output variables.\n",
    "\n",
    "\n",
    "While implementing the decision tree we will go through the following two phases:\n",
    "\n",
    "1. Building Phase\n",
    "    - Preprocess the dataset.\n",
    "    - Split the dataset from train and test using Python sklearn package.\n",
    "    - Train the classifier.\n",
    "2. Operational Phase\n",
    "    - Make predictions.\n",
    "    - Calculate the accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix \n",
    "from sklearn.cross_validation import train_test_split \n",
    "from sklearn.tree import DecisionTreeClassifier \n",
    "from sklearn.metrics import accuracy_score \n",
    "from sklearn.metrics import classification_report \n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess the dataset and preparing the data\n",
    "\n",
    "In this section we will divide our data into attributes and labels and will then divide the resultant data into both training and test sets. By doing this we can train our algorithm on one set of data and then test it out on a completely different set of data that the algorithm hasn't seen yet. This provides you with a more accurate view of how your trained algorithm will actually perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to split the dataset \n",
    "def splitdataset(df): \n",
    "  \n",
    "    # Seperating the target variable \n",
    "    X = df[['Likes','Page talking about','C1', 'C2', 'C3', 'C4','Base_Time','Post length', 'Monday', 'Tuesday',\n",
    "              'Wednesday', 'Thursday','Friday','Saturday', 'Sunday']]\n",
    "      \n",
    "    Y = df[['Class']]\n",
    "  \n",
    "    # Spliting the dataset into train and test \n",
    "    X_train, X_test, Y_train, Y_test = train_test_split(  \n",
    "    X, Y, test_size = 0.3, random_state = 100) \n",
    "      \n",
    "    return X, Y, X_train, X_test, Y_train, Y_test "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to scale our dataset. \n",
    "When your data is comprised of attributes with varying scales, many machine learning algorithms can benefit from rescaling the attributes to all have the same scale.\n",
    "\n",
    "Often this is referred to as normalization and attributes are often rescaled into the range between 0 and 1. This is useful for optimization algorithms in used in the core of machine learning algorithms like gradient descent. It is also useful for algorithms that weight inputs like regression and neural networks and algorithms that use distance measures like K-Nearest Neighbours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scale(x_train, x_test):\n",
    "    scaler = StandardScaler()  \n",
    "    scaler.fit(x_train)\n",
    "    x_train = scaler.transform(x_train)  \n",
    "    x_test = scaler.transform(x_test)\n",
    "    \n",
    "    return x_train, x_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the classifier\n",
    "In this section we are going to use gini index and information gain. Both of these methods are used to select from the n attributes of the dataset which attribute would be placed at the root node or the internal node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to perform training with giniIndex. \n",
    "def train_using_gini(X_train, X_test, Y_train): \n",
    "  \n",
    "    # Creating the classifier object \n",
    "    clf_gini = DecisionTreeClassifier(criterion = \"gini\", \n",
    "            random_state = 100,max_depth=3, min_samples_leaf=5) \n",
    "  \n",
    "    # Performing training \n",
    "    clf_gini.fit(X_train, Y_train) \n",
    "    return clf_gini "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to use entropy. Entropy controls how a Decision Tree decides to split the data. It actually effects how a Decision Tree draws its boundaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to perform training with entropy. \n",
    "def train_using_entropy(X_train, X_test, Y_train): \n",
    "  \n",
    "    # Decision tree with entropy \n",
    "    clf_entropy = DecisionTreeClassifier( \n",
    "            criterion = \"entropy\", random_state = 100, \n",
    "            max_depth = 3, min_samples_leaf = 5) \n",
    "  \n",
    "    # Performing training \n",
    "    clf_entropy.fit(X_train, Y_train) \n",
    "    return clf_entropy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make predictions\n",
    "In this section we are going to make predictions on test with giniIndex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to make predictions \n",
    "def prediction(X_test, clf_object): \n",
    "  \n",
    "    # Predicton on test with giniIndex \n",
    "    Y_pred = clf_object.predict(X_test) \n",
    "    print(\"Predicted values:\") \n",
    "    print(Y_pred) \n",
    "    return Y_pred "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate the accuracy\n",
    "In this section we are going to calculate the accuracy using the accuracy_score function. Accuracy score is used to calculate the accuracy of the trained classifier.\n",
    "Confusion matrix is calculated as well. Confusion Matrix is used to understand the trained classifier behavior over the test dataset or validate dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to calculate accuracy \n",
    "def cal_accuracy(Y_test, Y_pred): \n",
    "      \n",
    "    print(\"Confusion Matrix: \", \n",
    "        confusion_matrix(Y_test, Y_pred)) \n",
    "      \n",
    "    print (\"Accuracy : \", \n",
    "    accuracy_score(Y_test,Y_pred)*100) \n",
    "      \n",
    "    print(\"Report : \", \n",
    "    classification_report(Y_test, Y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Building Phase\n",
    "dfNew = create_class(df)\n",
    "X, Y, X_train, X_test, Y_train, Y_test = splitdataset(dfNew)\n",
    "X_train, X_test = scale(X_train, X_test)\n",
    "\n",
    "clf_gini = train_using_gini(X_train, X_test, Y_train) \n",
    "clf_entropy = train_using_entropy(X_train, X_test, Y_train) \n",
    "\n",
    "# Operational Phase \n",
    "print(\"Results Using Gini Index:\") \n",
    "\n",
    "# Prediction using gini \n",
    "Y_pred_gini = prediction(X_test, clf_gini) \n",
    "cal_accuracy(Y_test, Y_pred_gini) \n",
    "\n",
    "print(\"Results Using Entropy:\") \n",
    "# Prediction using entropy \n",
    "Y_pred_entropy = prediction(X_test, clf_entropy) \n",
    "cal_accuracy(Y_test, Y_pred_entropy) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use **Multi-layer Perceptron (MLP)** that is a supervised learning algorithm that learns a function by training on a dataset. Given a set of features X = x1,x2,...,xm and a target Y, it can learn a non-linear function approximator for either classification or regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import *\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfNew = create_class(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class MLPClassifier implements a multi-layer perceptron (MLP) algorithm that trains using Backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuralNetwork = MLPClassifier(hidden_layer_sizes=(100, 100), max_iter = 1000)\n",
    "\n",
    "X = dfNew[['Likes','Page talking about','C1', 'C2', 'C3', 'C4','Base_Time','Post length', 'Monday', 'Tuesday','Wednesday', 'Thursday','Friday','Saturday', 'Sunday']] \n",
    "Y = dfNew[[\"Class\"]]\n",
    "\n",
    "Y = pd.get_dummies(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train and test sets splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.1)\n",
    "X_train, X_test = scale(X_train, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural network's training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuralNetwork.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After fitting (training), the model can predict labels for new samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = neuralNetwork.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to calculate the accuracy using the accuracy_score function and the loss using the zero_one_loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(Y_test, Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zero_one_loss(Y_test, Y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that Decision Tree and Neural Network achieved ***a mean classification accuracy of about 78%***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working on the dataset, the goal we set ourselves, that is to see how much influence a short post (with a number of characters less than 400) compared to a long one (with a number of characters more than 400) has, has been achieved by our analysis thanks to the development of various phases.\n",
    "   The first has been the one referred ***to clean the dataset***, *eliminating all those columns that were repetitive or empty and checking that all numerical values within it were good and not corrupted*, in order to conduct a correct analysis.\n",
    "   Successively has been possible to go through the ***exploratory analysis***. This showed us that for every week-day, at most, *a number close to 6000 posts are published*. ***The day when there are more publications is Thursday***, so exactly in the middle of the week, while ***the day when there are fewer publications is Monday***, with a number close to 5000 posts.\n",
    "Analysis of the comments was carried out, which showed that ***the period in which more comments are made and more likes are placed is between 48 and 72 hours after the publication of a post***.\n",
    "An analysis was also carried out on the number of likes that a post receives on average, from which it was found that more than 7000 posts, out of a total of 40948, receive less than 400.\n",
    "Another analysis that was carried out was about categories. In particular, has been analysed ***the category that shows the most interactions, which was found to be 'Professional Sports Team'***.\n",
    "    After analysing all these 'secondary' quantities that are related to what is our main purpose, the final analysis was initiated and completed. To do this, the posts with more than 400 characters were initially separated from those with less than the same number, identifying them respectively as 'Long Posts' and 'Short Posts'. This showed that about 90% of posts have less than 400 characters. This already mean that a post defined as short is certainly preferred over one defined as long. \n",
    "Referring to this, it was also possible to note that the former is shared about 90% more.\n",
    "Finally, deepening the study, it was noticed that this is done more in the ***24 hours following the sharing of the post for short posts***, while ***between 24h and 48h for long ones***.\n",
    "In addition to the number of shares, the number of comments received from these two types of posts was also analysed.\n",
    "The results were almost identical, clearly showing that short length posts are:\n",
    "- ***preferred by users*** ,\n",
    "- ***have more interactions***.\n",
    "\n",
    "For this phase we have also tried to give an ***'ideal' number of characters to obtain the largest possible number of iterations (represented by likes, shares and comments) which was found to be about 107***.\n",
    "Finally, ***supervised and unsupervised learning*** were performed.\n",
    "In the first case, two different Machine Learning prediction methodologies have been implemented:\n",
    "-  'Decision Tree' and\n",
    "-  'Neural Network', \n",
    "\n",
    "both with the aim of predicting the number of shares that a post will have by identifying 4 different classes:\n",
    "- A low number of shares that indicates a number between 1 and 49 and it is identified by the number '1'\n",
    "- A medium number of shares that indicates a number between 50 and 199 and it is identified by the number '2'\n",
    "- A good number of shares that indicates a number between 200 and 699 and it is identified by the number '3'\n",
    "- An high number of shares that indicates a number grater then 700 and it is identified by the number '4'\n",
    "\n",
    "In both cases, an ***accuracy level close to 78% were achieved***.\n",
    "Concerning the Unsupervised learning, we adopted a ***clustering algorithm called K-Means Clustering*** that clustered our data points into a number (8) of mutually exclusive clusters. After determining the right number of clusters through the elbow method, we interpreted the 8 cluster solution and plotted them.  \n",
    "The resulting scatterplot shows that the seven clusters excluding the one more spread out are dense pack meaning that the observation within the clusters are pretty highly correlated between each other and within cluster's variance is relative low. This overlap means that there is not good separation between these seven clusters. On the other hand, the cluster with centre at position around (8, -0.7) shows a better separation and the observations are more spread out indicating less correlation among the observations and high within cluster's variance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}