01. Overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Overview\n",
    "\n",
    "[Data Science Playlist on YouTube](https://www.youtube.com/watch?v=r_LB_k17MGE&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy)\n",
    "[![Python Data Science](https://apmonitor.com/che263/uploads/Begin_Python/DataScience01.png)](https://www.youtube.com/watch?v=r_LB_k17MGE&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy \"Python Data Science\")\n",
    "\n",
    "### Introduction\n",
    "\n",
    "Welcome to the course on data science with Python. This course steps through basic data science and machine learning skills to analyze data and create actionable information. It address major steps of the Cross-Industry Standard Process for Data Mining Cycle (CRISP-DM) that has 6 phases.\n",
    "\n",
    "1. Business Understanding - purpose of the data science project\n",
    "2. Data Understanding - assess available data to address the business case\n",
    "3. Data Preparation - exploratory data analysis, feature engineering, and transformations \n",
    "4. Modeling - select and train regressors or classifiers\n",
    "5. Evaluation - test models with various criteria\n",
    "6. Deployment - production phase to deploy models in data pipelines \n",
    "\n",
    "The 12 exercises are designed to be completed in 2-3 hours (15-20 minutes each) but sections can be skipped if you already have the background knowledge.\n",
    "\n",
    "1. [Overview (this lesson)](https://github.com/APMonitor/data_science/blob/master/01.%20Overview.ipynb)\n",
    "2. [Data Import and Export](https://github.com/APMonitor/data_science/blob/master/02.%20Import_Export.ipynb)\n",
    "3. [Data Analysis](https://github.com/APMonitor/data_science/blob/master/03.%20Analyze.ipynb)\n",
    "4. [Visualize Data](https://github.com/APMonitor/data_science/blob/master/04.%20Visualize.ipynb)\n",
    "5. [Prepare (Cleanse, Scale, Divide) Data](https://github.com/APMonitor/data_science/blob/master/05.%20Prepare_data.ipynb)\n",
    "6. [Regression](https://github.com/APMonitor/data_science/blob/master/06.%20Regression.ipynb)\n",
    "7. [Features](https://github.com/APMonitor/data_science/blob/master/07.%20Features.ipynb)\n",
    "8. [Classification](https://github.com/APMonitor/data_science/blob/master/08.%20Classification.ipynb)\n",
    "9. [Interpolation](https://github.com/APMonitor/data_science/blob/master/09.%20Interpolation.ipynb)\n",
    "10. [Solve Equations](https://github.com/APMonitor/data_science/blob/master/10.%20Solve_Equations.ipynb)\n",
    "11. [Differential Equations](https://github.com/APMonitor/data_science/blob/master/11.%20Differential_Equations.ipynb)\n",
    "12. [Time Series](https://github.com/APMonitor/data_science/blob/master/12.%20Time_Series.ipynb)\n",
    "\n",
    "It is best to follow the lessons in these steps because the later lessons build upon the information from the prior lessons.\n",
    "\n",
    "### Final Project\n",
    "\n",
    "![cell](https://apmonitor.com/che263/uploads/Begin_Python/cell.png)\n",
    "\n",
    "You are designing a next-generation cell phone and the battery and processor on the cell phone generate a lot of heat. You want to make sure that the material between them will prevent over-heating of the battery by the processor.\n",
    "\n",
    "![battery_cpu](https://apmonitor.com/che263/uploads/Begin_Python/battery_cpu.png)\n",
    "\n",
    "The final project will help you answer questions about material properties for predicting the temperature of the battery and processor. It uses data from the temperature control lab to determine thermal conductivity. See the [Final Project](https://github.com/APMonitor/data_science/blob/master/04.%20Visualize.ipynb) after going through the 12 lessons and when you are ready for the challenge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What Runs Python Commands?\n",
    "\n",
    "The Python executable runs the commands. A Jupyter notebook starts a Python session called a kernel. Restarting the kernel removes any of the prior cell results and stops any running code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where is the Current Working Directory?\n",
    "\n",
    "The current working directory is where the IPython notebook is located. It is also where files will be stored and retrieved. It is possible to change the current working directory with `os.chdir(path)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "print(os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are seperate Jupyter notebook files to help with TCLab installation. There are also [Frequently Asked Questions](https://apmonitor.com/pdc/index.php/Main/ArduinoSetup) for setup and troubleshooting. More information on installation of TCLab package is in the IPython notebook __TCLab Help__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![exercise](https://apmonitor.com/che263/uploads/Begin_Python/exercise.png)\n",
    "\n",
    "### Python Package Management\n",
    "\n",
    "Package management in data science and machine learning is an important topic because of the pace of new releases. It may be important to have the latest or a specific version of a package.\n",
    "\n",
    "To install packages, use the `pip` (or `conda`) package manager. An alternative is to start an Anaconda prompt from the start menu, start a DOS prompt with Windows-key+r `cmd`, or terminal on MacOS/Linux and type:\n",
    "\n",
    "![gekko](https://apmonitor.com/che263/uploads/Begin_Python/gekko.png)\n",
    "\n",
    "```\n",
    "python -m pip install gekko\n",
    "```\n",
    "\n",
    "If you would like to run this same command in a Jupyter notebook then use an exclamation mark in front.\n",
    "\n",
    "```python\n",
    "!python -m pip install gekko\n",
    "```\n",
    "\n",
    "If you want to be sure that you are using the distribution of the current Python kernel use `sys.executable`.\n",
    "\n",
    "```python\n",
    "import sys\n",
    "!{sys.executable} -m pip install gekko\n",
    "```\n",
    "\n",
    "Packages used in this course include `gekko`, `keras`, `numpy`, `pandas`, `pyserial`, `sklearn`, `tclab`, `tensorflow`, and others. If there is an error that you are missing a package then `!pip install gekko` is typically sufficient to retrieve and install it. The `!` tells Jupyter to run from the system command line, not as a Python command although the `!` is now optional for `pip` in the latest version of Jupyter. Add `--user` (not your specific user name) if you do not have administrative privilege on the computer. The package manager `pip` retrieves the [lastest version of gekko](https://pypi.org/project/gekko/) from `pypi` and installs it in the Python distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Install Package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install --user gekko"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### List Available Packages and Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Uninstall Package\n",
    "\n",
    "If you need to `uninstall` a package then use the command line because it will ask for a `yes`/`no` response or else add `--yes` to automatically answer the confirmation message. Don't forget to reinstall the package if you need it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip uninstall gekko --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Install Specific Version\n",
    "\n",
    "You may need to get a specific package version by specifying a version such as `0.2.0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install gekko==0.2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Upgrade Package\n",
    "\n",
    "If you have an older version and want to upgrade to the latest stable release use `--upgrade`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install gekko --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![list](https://apmonitor.com/che263/uploads/Begin_Python/list.png)\n",
    "\n",
    "### `pip` Summary\n",
    "\n",
    "The `pip` tool facilitates package management in Python using `install`, `list`, and `uninstall` commands.\n",
    "\n",
    "```python\n",
    "!python -m pip {command} {package name}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Install TCLab Module\n",
    "\n",
    "You need the TCLab to do the exercises but there are sample data files if you do not have a device. The [TCLab](https://apmonitor.com/heat.htm) is available online with [Amazon](https://www.amazon.com/TCLab-Temperature-Control-Lab/dp/B07GMFWMRY) or with a [few other options](https://apmonitor.com/pdc/index.php/Main/PurchaseLabKit).\n",
    "\n",
    "![connections_no_power](https://apmonitor.com/che263/uploads/Begin_Python/connections_no_power.png)\n",
    "\n",
    "As a first step, plug in the TCLab (USB blue cable only) and install the package with `python -m pip install tclab` or by running the cell below (`Ctrl+Enter`). Restart the Python kernel with `Kernel...Restart & Run All` from the menu if there is an error importing `tclab` after the installation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install tclab\n",
    "try:\n",
    "    import tclab\n",
    "except:\n",
    "    # Needed to communicate through usb port\n",
    "    !python -m pip install --user pyserial\n",
    "    # The --user is put in for accounts without admin privileges\n",
    "    !python -m pip install --user tclab \n",
    "    # restart kernel if this doesn't import\n",
    "    try:\n",
    "        import tclab\n",
    "    except:\n",
    "        print('Restart kernel from menu if Dead kernel')\n",
    "        print('Restart kernel automatically...')\n",
    "        import IPython\n",
    "        app = IPython.Application.instance()\n",
    "        app.kernel.do_shutdown(restart=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TCLab LED Test\n",
    "\n",
    "The following code show how to turn on the LED for 5 seconds. You only need to connect the blue USB cable to the computer. The white power cable is needed for heater power in the next example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TCLab version 0.4.9\n",
      "Arduino Leonardo connected on port COM8 at 115200 baud.\n",
      "TCLab Firmware Version 1.01.\n",
      "TCLab disconnected successfully.\n"
     ]
    }
   ],
   "source": [
    "import tclab\n",
    "import time\n",
    "\n",
    "with tclab.TCLab() as lab:\n",
    "    lab.LED(100)\n",
    "    time.sleep(5.0) # wait 5.0 seconds\n",
    "    lab.LED(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TCLab Heater Test\n",
    "\n",
    "![connections](https://apmonitor.com/che263/uploads/Begin_Python/connections.png)\n",
    "\n",
    "The following code show how to turn on the heater `Q1`. It reads temperature `T1` before and after a 10 second pause. You need to connect the white power cable and plug in the power supply. The white power cable is needed to supply power to the heaters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TCLab version 0.4.9\n",
      "Arduino Leonardo connected on port COM8 at 115200 baud.\n",
      "TCLab Firmware Version 1.01.\n",
      "22.83\n",
      "22.83\n",
      "TCLab disconnected successfully.\n"
     ]
    }
   ],
   "source": [
    "with tclab.TCLab() as lab:\n",
    "    print(lab.T1)  # print temperature 1\n",
    "    lab.Q1(100)    # turn on Q1 to 100%\n",
    "    time.sleep(15) # sleep for 15 seconds\n",
    "    print(lab.T1)  # print temperature 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![expert](https://apmonitor.com/che263/uploads/Begin_Python/expert.png)\n",
    "\n",
    "### TCLab Activity\n",
    "\n",
    "TCLab functions allow Python to interact with the TCLab through the USB serial connection.\n",
    "\n",
    "| **TCLab Function** | **Example** | **Description** |\n",
    "| ----------- | ----------- | ----------- |\n",
    "| `TCLab()` | `tclab.TCLab()` | Create new lab object and connect |\n",
    "| `LED` | `lab.LED(45)` | Turn on the LED to 45%. Valid range is 0-100% |\n",
    "| `Q1` | `lab.Q1(63)` | Turn on heater 1 (`Q1`) to 63%. Valid range is 0-100% |\n",
    "| `Q2` | `lab.Q2(28)` | Turn on heater 2 (`Q2`) to 28%. Valid range is 0-100% |\n",
    "| `T1` | `print(lab.T1)` | Read temperature 1 (`T1`) in °C. valid range is -40 to 150°C (TMP36 sensor) |\n",
    "| `T2` | `print(lab.T2)` | Read temperature 2 (`T2`) in °C. with +/- 1°C accuracy (TMP36 sensor) |\n",
    "| `close()` | `lab.close()` | Close serial USB connection to TCLab - not needed if using `with` to open |\n",
    "\n",
    "Complete the following exercises.\n",
    "\n",
    "### Blink LED\n",
    "\n",
    "Blink the LED 5 times for 1 second each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Heat to 50°C\n",
    "\n",
    "Turn on heater 1 (`Q1`) to 80% until `T1` reaches 50°C. Update the LED blink time (`t`). \n",
    "\n",
    "$t = \\frac{50-T_1}{10}$\n",
    "\n",
    "This displays a visual indication of the temperature (`T1`) with the formula for alternating time off and time on. As the temperature approaches 50°C, the LED blinks more rapidly until it turns off.\n",
    "\n",
    "Use a `while lab.T1<50:` loop to continuously check that the temperature is less than 50°C."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}