Linear_Regression_Consulting_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Consulting Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Congratulations! You've been contracted by Hyundai Heavy Industries to help them build a predictive model for some ships. [Hyundai Heavy Industries](http://www.hyundai.eu/en) is one of the world's largest ship manufacturing companies and builds cruise liners.\n",
    "\n",
    "You've been flown to their headquarters in Ulsan, South Korea to help them give accurate estimates of how many crew members a ship will require.\n",
    "\n",
    "They are currently building new ships for some customers and want you to create a model and use it to predict how many crew members the ships will need.\n",
    "\n",
    "Here is what the data looks like so far:\n",
    "\n",
    "    Description: Measurements of ship size, capacity, crew, and age for 158 cruise\n",
    "    ships.\n",
    "\n",
    "\n",
    "    Variables/Columns\n",
    "    Ship Name     1-20\n",
    "    Cruise Line   21-40\n",
    "    Age (as of 2013)   46-48\n",
    "    Tonnage (1000s of tons)   50-56\n",
    "    passengers (100s)   58-64\n",
    "    Length (100s of feet)  66-72\n",
    "    Cabins  (100s)   74-80\n",
    "    Passenger Density   82-88\n",
    "    Crew  (100s)   90-96\n",
    "    \n",
    "It is saved in a csv file for you called \"cruise_ship_info.csv\". Your job is to create a regression model that will help predict how many crew members will be needed for future ships. The client also mentioned that they have found that particular cruise lines will differ in acceptable crew counts, so it is most likely an important feature to include in your analysis! \n",
    "\n",
    "Once you've created the model and tested it for a quick check on how well you can expect it to perform, make sure you take a look at why it performs so well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.feature import VectorAssembler, StringIndexer\n",
    "from pyspark.sql.functions import corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://DESKTOP-BJS1Q67:4041\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.5.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>lr_consulting_project</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x1fd23aefa10>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark = SparkSession.builder.appName('lr_consulting_project').getOrCreate()\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Ship_name: string (nullable = true)\n",
      " |-- Cruise_line: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- Tonnage: double (nullable = true)\n",
      " |-- passengers: double (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- cabins: double (nullable = true)\n",
      " |-- passenger_density: double (nullable = true)\n",
      " |-- crew: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.read.csv('cruise_ship_info.csv',inferSchema=True,header=True)\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "158"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|summary|Ship_name|Cruise_line|               Age|           Tonnage|       passengers|           length|            cabins|passenger_density|             crew|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|  count|      158|        158|               158|               158|              158|              158|               158|              158|              158|\n",
      "|   mean| Infinity|       NULL|15.689873417721518| 71.28467088607599|18.45740506329114|8.130632911392404| 8.830000000000005|39.90094936708861|7.794177215189873|\n",
      "| stddev|     NULL|       NULL| 7.615691058751413|37.229540025907866|9.677094775143416|1.793473548054825|4.4714172221480615| 8.63921711391542|3.503486564627034|\n",
      "|    min|Adventure|    Azamara|                 4|             2.329|             0.66|             2.79|              0.33|             17.7|             0.59|\n",
      "|    max|Zuiderdam|   Windstar|                48|             220.0|             54.0|            11.82|              27.0|            71.43|             21.0|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+-----+\n",
      "|      Cruise_line|count|\n",
      "+-----------------+-----+\n",
      "|            Costa|   11|\n",
      "|              P&O|    6|\n",
      "|           Cunard|    3|\n",
      "|Regent_Seven_Seas|    5|\n",
      "|              MSC|    8|\n",
      "|         Carnival|   22|\n",
      "|          Crystal|    2|\n",
      "|           Orient|    1|\n",
      "|         Princess|   17|\n",
      "|        Silversea|    4|\n",
      "|         Seabourn|    3|\n",
      "| Holland_American|   14|\n",
      "|         Windstar|    3|\n",
      "|           Disney|    2|\n",
      "|        Norwegian|   13|\n",
      "|          Oceania|    3|\n",
      "|          Azamara|    2|\n",
      "|        Celebrity|   10|\n",
      "|             Star|    6|\n",
      "|  Royal_Caribbean|   23|\n",
      "+-----------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.groupBy('Cruise_line').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StringIndexer_52cd8c22dcb4"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexer = StringIndexer(inputCol='Cruise_line',outputCol='Cruise_line_indexed')\n",
    "indexer.setHandleInvalid(\"error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StringIndexerModel: uid=StringIndexer_52cd8c22dcb4, handleInvalid=error"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexer_model = indexer.fit(df)\n",
    "indexer_model.setHandleInvalid(\"error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[Ship_name: string, Cruise_line: string, Age: int, Tonnage: double, passengers: double, length: double, cabins: double, passenger_density: double, crew: double, Cruise_line_indexed: double]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = indexer_model.transform(df)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|Cruise_line_indexed|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|               16.0|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|               16.0|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|                1.0|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|                1.0|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|                1.0|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|                1.0|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|                1.0|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|                1.0|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|                1.0|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|                1.0|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|                1.0|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|                1.0|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|                1.0|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+-------------------+\n",
      "|summary|Ship_name|Cruise_line|               Age|           Tonnage|       passengers|           length|            cabins|passenger_density|             crew|Cruise_line_indexed|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+-------------------+\n",
      "|  count|      158|        158|               158|               158|              158|              158|               158|              158|              158|                158|\n",
      "|   mean| Infinity|       NULL|15.689873417721518| 71.28467088607599|18.45740506329114|8.130632911392404| 8.830000000000005|39.90094936708861|7.794177215189873|  5.063291139240507|\n",
      "| stddev|     NULL|       NULL| 7.615691058751413|37.229540025907866|9.677094775143416|1.793473548054825|4.4714172221480615| 8.63921711391542|3.503486564627034|  4.758744608182735|\n",
      "|    min|Adventure|    Azamara|                 4|             2.329|             0.66|             2.79|              0.33|             17.7|             0.59|                0.0|\n",
      "|    max|Zuiderdam|   Windstar|                48|             220.0|             54.0|            11.82|              27.0|            71.43|             21.0|               19.0|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ship_name',\n",
       " 'Cruise_line',\n",
       " 'Age',\n",
       " 'Tonnage',\n",
       " 'passengers',\n",
       " 'length',\n",
       " 'cabins',\n",
       " 'passenger_density',\n",
       " 'crew',\n",
       " 'Cruise_line_indexed']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VectorAssembler_5dd44feb4001"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assembler = VectorAssembler(inputCols=[\n",
    "     'Age',\n",
    "     'Tonnage',\n",
    "     'passengers',\n",
    "     'length',\n",
    "     'cabins',\n",
    "     'passenger_density',\n",
    "     'Cruise_line_indexed'\n",
    "],outputCol='features')\n",
    "assembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+--------------------+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|Cruise_line_indexed|            features|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+--------------------+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|               16.0|[6.0,30.276999999...|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|               16.0|[6.0,30.276999999...|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|                1.0|[26.0,47.262,14.8...|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|                1.0|[11.0,110.0,29.74...|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|                1.0|[17.0,101.353,26....|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[22.0,70.367,20.5...|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[15.0,70.367,20.5...|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|                1.0|[23.0,70.367,20.5...|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[19.0,70.367,20.5...|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|                1.0|[6.0,110.23899999...|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|                1.0|[10.0,110.0,29.74...|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|                1.0|[28.0,46.052,14.5...|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[18.0,70.367,20.5...|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[17.0,70.367,20.5...|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|                1.0|[11.0,86.0,21.24,...|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|                1.0|[8.0,110.0,29.74,...|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|                1.0|[9.0,88.5,21.24,9...|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[15.0,70.367,20.5...|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|                1.0|[12.0,88.5,21.24,...|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|                1.0|[20.0,70.367,20.5...|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "assembled_df = assembler.transform(output)\n",
    "assembled_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Ship_name: string (nullable = true)\n",
      " |-- Cruise_line: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- Tonnage: double (nullable = true)\n",
      " |-- passengers: double (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- cabins: double (nullable = true)\n",
      " |-- passenger_density: double (nullable = true)\n",
      " |-- crew: double (nullable = true)\n",
      " |-- Cruise_line_indexed: double (nullable = false)\n",
      " |-- features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "assembled_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Ship_name='Journey', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, Cruise_line_indexed=16.0, features=DenseVector([6.0, 30.277, 6.94, 5.94, 3.55, 42.64, 16.0]))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assembled_df.head(1)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features|crew|\n",
      "+--------------------+----+\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[26.0,47.262,14.8...| 6.7|\n",
      "|[11.0,110.0,29.74...|19.1|\n",
      "|[17.0,101.353,26....|10.0|\n",
      "|[22.0,70.367,20.5...| 9.2|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[23.0,70.367,20.5...| 9.2|\n",
      "|[19.0,70.367,20.5...| 9.2|\n",
      "|[6.0,110.23899999...|11.5|\n",
      "|[10.0,110.0,29.74...|11.6|\n",
      "|[28.0,46.052,14.5...| 6.6|\n",
      "|[18.0,70.367,20.5...| 9.2|\n",
      "|[17.0,70.367,20.5...| 9.2|\n",
      "|[11.0,86.0,21.24,...| 9.3|\n",
      "|[8.0,110.0,29.74,...|11.6|\n",
      "|[9.0,88.5,21.24,9...|10.3|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[12.0,88.5,21.24,...| 9.3|\n",
      "|[20.0,70.367,20.5...| 9.2|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_df = assembled_df.select('features','crew')\n",
    "final_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "158"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----------------+\n",
      "|summary|             crew|\n",
      "+-------+-----------------+\n",
      "|  count|              158|\n",
      "|   mean|7.794177215189873|\n",
      "| stddev|3.503486564627034|\n",
      "|    min|             0.59|\n",
      "|    max|             21.0|\n",
      "+-------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+\n",
      "|            features| crew|\n",
      "+--------------------+-----+\n",
      "|[5.0,86.0,21.04,9...|  8.0|\n",
      "|[5.0,115.0,35.74,...| 12.2|\n",
      "|[5.0,122.0,28.5,1...|  6.7|\n",
      "|[5.0,133.5,39.59,...|13.13|\n",
      "|[5.0,160.0,36.34,...| 13.6|\n",
      "|[6.0,30.276999999...| 3.55|\n",
      "|[6.0,90.0,20.0,9....|  9.0|\n",
      "|[6.0,93.0,23.94,9...|11.09|\n",
      "|[6.0,112.0,38.0,9...| 10.9|\n",
      "|[6.0,113.0,37.82,...| 12.0|\n",
      "|[6.0,158.0,43.7,1...| 13.6|\n",
      "|[7.0,89.6,25.5,9....| 9.87|\n",
      "|[7.0,158.0,43.7,1...| 13.6|\n",
      "|[8.0,77.499,19.5,...|  9.0|\n",
      "|[8.0,91.0,22.44,9...| 11.0|\n",
      "|[8.0,110.0,29.74,...| 11.6|\n",
      "|[9.0,59.058,17.0,...|  7.4|\n",
      "|[9.0,81.0,21.44,9...| 10.0|\n",
      "|[9.0,85.0,19.68,9...| 8.69|\n",
      "|[9.0,88.5,21.24,9...| 10.3|\n",
      "+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_df, test_df = final_df.randomSplit([0.7,0.3])\n",
    "train_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+\n",
      "|            features| crew|\n",
      "+--------------------+-----+\n",
      "|[4.0,220.0,54.0,1...| 21.0|\n",
      "|[6.0,30.276999999...| 3.55|\n",
      "|[6.0,110.23899999...| 11.5|\n",
      "|[7.0,116.0,31.0,9...| 12.0|\n",
      "|[10.0,77.0,20.16,...|  9.0|\n",
      "|[10.0,81.76899999...| 8.42|\n",
      "|[10.0,110.0,29.74...| 11.6|\n",
      "|[11.0,138.0,31.14...|11.85|\n",
      "|[12.0,77.104,20.0...| 9.59|\n",
      "|[13.0,25.0,3.82,5...| 2.95|\n",
      "|[13.0,91.0,20.32,...| 9.99|\n",
      "|[14.0,33.0,4.9,5....| 3.24|\n",
      "|[14.0,76.8,19.6,8...| 12.0|\n",
      "|[14.0,83.0,17.5,9...| 9.45|\n",
      "|[15.0,30.27699999...|  4.0|\n",
      "|[17.0,70.0,20.76,...|  7.2|\n",
      "|[18.0,70.367,20.5...|  9.2|\n",
      "|[18.0,70.60600000...| 8.58|\n",
      "|[18.0,77.499,19.5...|  9.0|\n",
      "|[19.0,16.8,2.96,5...|  2.1|\n",
      "+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+\n",
      "|summary|              crew|\n",
      "+-------+------------------+\n",
      "|  count|               119|\n",
      "|   mean| 7.974789915966397|\n",
      "| stddev|3.3580045650475965|\n",
      "|    min|              0.59|\n",
      "|    max|              19.1|\n",
      "+-------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+\n",
      "|summary|              crew|\n",
      "+-------+------------------+\n",
      "|  count|                39|\n",
      "|   mean| 7.243076923076919|\n",
      "| stddev|3.9093882640625868|\n",
      "|    min|              0.88|\n",
      "|    max|              21.0|\n",
      "+-------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression_e9926ff89384"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LinearRegression(featuresCol='features',labelCol='crew',predictionCol='prediction')\n",
    "lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyspark.ml.regression.LinearRegressionTrainingSummary at 0x1fd255fc2d0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model = lr.fit(train_df)\n",
    "lr_model.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DenseVector([-0.0209, 0.0154, -0.1562, 0.3422, 0.8343, -0.0094, 0.0528])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.1612990321896567"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.numFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Age',\n",
       " 'Tonnage',\n",
       " 'passengers',\n",
       " 'length',\n",
       " 'cabins',\n",
       " 'passenger_density',\n",
       " 'Cruise_line_indexed']"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assembler.getInputCols()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Age': -0.02089828517729866,\n",
       " 'Tonnage': 0.015443430983301597,\n",
       " 'passengers': -0.1561866947500571,\n",
       " 'length': 0.3421716648426525,\n",
       " 'cabins': 0.8343037280140753,\n",
       " 'passenger_density': -0.009443717243607725,\n",
       " 'Cruise_line_indexed': 0.052776167963893186}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_imps = dict(zip(assembler.getInputCols(),lr_model.coefficients))\n",
    "feature_imps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('cabins', 0.8343037280140753),\n",
       " ('length', 0.3421716648426525),\n",
       " ('Cruise_line_indexed', 0.052776167963893186),\n",
       " ('Tonnage', 0.015443430983301597),\n",
       " ('passenger_density', -0.009443717243607725),\n",
       " ('Age', -0.02089828517729866),\n",
       " ('passengers', -0.1561866947500571)]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_imps = sorted(feature_imps.items(),key=lambda kv: (kv[1],kv[0]),reverse=True)\n",
    "feature_imps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the model coefficients, we can infer that the feature 'cabin' is the most significant when it comes to predicting the number of crew members in a particular ship."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyspark.ml.regression.LinearRegressionSummary at 0x1fd256609d0>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results = lr_model.evaluate(test_df)\n",
    "test_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9603810093776035"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.r2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9514347856886752"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.r2adj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+------------------+\n",
      "|            features| crew|        prediction|\n",
      "+--------------------+-----+------------------+\n",
      "|[4.0,220.0,54.0,1...| 21.0|20.904513821240023|\n",
      "|[6.0,30.276999999...| 3.55| 4.532972862832758|\n",
      "|[6.0,110.23899999...| 11.5|11.068468738662364|\n",
      "|[7.0,116.0,31.0,9...| 12.0| 12.63839344288523|\n",
      "|[10.0,77.0,20.16,...|  9.0| 8.795143116958535|\n",
      "|[10.0,81.76899999...| 8.42|  8.73701863368302|\n",
      "|[10.0,110.0,29.74...| 11.6|12.047105257679602|\n",
      "|[11.0,138.0,31.14...|11.85|12.938074110376412|\n",
      "|[12.0,77.104,20.0...| 9.59|  8.76934595865606|\n",
      "|[13.0,25.0,3.82,5...| 2.95| 2.980236486609423|\n",
      "|[13.0,91.0,20.32,...| 9.99|  9.12884710614818|\n",
      "|[14.0,33.0,4.9,5....| 3.24|3.1423761735371247|\n",
      "|[14.0,76.8,19.6,8...| 12.0| 8.851307911678116|\n",
      "|[14.0,83.0,17.5,9...| 9.45| 9.195410572508287|\n",
      "|[15.0,30.27699999...|  4.0| 4.078420155243922|\n",
      "|[17.0,70.0,20.76,...|  7.2| 7.495640321034496|\n",
      "|[18.0,70.367,20.5...|  9.2|  8.50870563017861|\n",
      "|[18.0,70.60600000...| 8.58| 7.817230041076574|\n",
      "|[18.0,77.499,19.5...|  9.0| 8.407451556624899|\n",
      "|[19.0,16.8,2.96,5...|  2.1|2.2768149051086826|\n",
      "+--------------------+-----+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.298997786135132"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.explainedVariance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7681039580890837"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5489336583992321"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.meanAbsoluteError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.589983690432117"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.meanSquaredError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|           residuals|\n",
      "+--------------------+\n",
      "| 0.09548617875997678|\n",
      "| -0.9829728628327583|\n",
      "| 0.43153126133763564|\n",
      "| -0.6383934428852296|\n",
      "|  0.2048568830414652|\n",
      "|-0.31701863368301986|\n",
      "|-0.44710525767960263|\n",
      "| -1.0880741103764127|\n",
      "|  0.8206540413439392|\n",
      "|-0.03023648660942...|\n",
      "|  0.8611528938518198|\n",
      "| 0.09762382646287548|\n",
      "|  3.1486920883218836|\n",
      "| 0.25458942749171243|\n",
      "| -0.0784201552439221|\n",
      "|-0.29564032103449556|\n",
      "|  0.6912943698213887|\n",
      "|  0.7627699589234265|\n",
      "|  0.5925484433751009|\n",
      "|-0.17681490510868247|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.residuals.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.degreesOfFreedom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1.1222501704262848, 3.1486920883218836]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.devianceResiduals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "39"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.numInstances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|[4.0,220.0,54.0,1...|\n",
      "|[6.0,30.276999999...|\n",
      "|[6.0,110.23899999...|\n",
      "|[7.0,116.0,31.0,9...|\n",
      "|[10.0,77.0,20.16,...|\n",
      "|[10.0,81.76899999...|\n",
      "|[10.0,110.0,29.74...|\n",
      "|[11.0,138.0,31.14...|\n",
      "|[12.0,77.104,20.0...|\n",
      "|[13.0,25.0,3.82,5...|\n",
      "|[13.0,91.0,20.32,...|\n",
      "|[14.0,33.0,4.9,5....|\n",
      "|[14.0,76.8,19.6,8...|\n",
      "|[14.0,83.0,17.5,9...|\n",
      "|[15.0,30.27699999...|\n",
      "|[17.0,70.0,20.76,...|\n",
      "|[18.0,70.367,20.5...|\n",
      "|[18.0,70.60600000...|\n",
      "|[18.0,77.499,19.5...|\n",
      "|[19.0,16.8,2.96,5...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_df = test_df.select('features')\n",
    "unlabeled_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------------------+\n",
      "|            features|        prediction|\n",
      "+--------------------+------------------+\n",
      "|[4.0,220.0,54.0,1...|20.904513821240023|\n",
      "|[6.0,30.276999999...| 4.532972862832758|\n",
      "|[6.0,110.23899999...|11.068468738662364|\n",
      "|[7.0,116.0,31.0,9...| 12.63839344288523|\n",
      "|[10.0,77.0,20.16,...| 8.795143116958535|\n",
      "|[10.0,81.76899999...|  8.73701863368302|\n",
      "|[10.0,110.0,29.74...|12.047105257679602|\n",
      "|[11.0,138.0,31.14...|12.938074110376412|\n",
      "|[12.0,77.104,20.0...|  8.76934595865606|\n",
      "|[13.0,25.0,3.82,5...| 2.980236486609423|\n",
      "|[13.0,91.0,20.32,...|  9.12884710614818|\n",
      "|[14.0,33.0,4.9,5....|3.1423761735371247|\n",
      "|[14.0,76.8,19.6,8...| 8.851307911678116|\n",
      "|[14.0,83.0,17.5,9...| 9.195410572508287|\n",
      "|[15.0,30.27699999...| 4.078420155243922|\n",
      "|[17.0,70.0,20.76,...| 7.495640321034496|\n",
      "|[18.0,70.367,20.5...|  8.50870563017861|\n",
      "|[18.0,70.60600000...| 7.817230041076574|\n",
      "|[18.0,77.499,19.5...| 8.407451556624899|\n",
      "|[19.0,16.8,2.96,5...|2.2768149051086826|\n",
      "+--------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_predictions = lr_model.transform(unlabeled_df)\n",
    "test_predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Linear regression model performs really well as it has achieved an amazing r2 score of 96% on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|corr(cabins, crew)|\n",
      "+------------------+\n",
      "|0.9508226063578497|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr(col1='cabins',col2='crew')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+\n",
      "|corr(passengers, crew)|\n",
      "+----------------------+\n",
      "|    0.9152341306065384|\n",
      "+----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr(col1='passengers',col2='crew')).show()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}