mission_to_mars.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1 - Scraping\n",
    "1- Go to NASA Mars News Site (https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. \n",
    "Assign the text to variables that you can reference later.\n",
    "\n",
    "2- JPL Mars Space Images - Featured Image\n",
    "Visit the url for JPL Featured Space Image (https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars). Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called featured_image_url. \n",
    "Make sure to find the image url to the full size .jpg image. \n",
    "Make sure to save a complete url string for this image.\n",
    "\n",
    "3-Mars Weather Visit the Mars Weather twitter account (https://twitter.com/marswxreport?lang=en) and scrape the latest Mars weather tweet from the page. \n",
    "Save the tweet text for the weather report as a variable called mars_weather.\n",
    "\n",
    "4-Mars Facts Visit the Mars Facts webpage (https://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc. \n",
    "Use Pandas to convert the data to a HTML table string.\n",
    "\n",
    "5-Mars Hemispheres Visit the USGS Astrogeology site (https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres. \n",
    "You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.\n",
    "Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. \n",
    "Use a Python dictionary to store the data using the keys img_url and title. \n",
    "Append the dictionary with the image url string and the hemisphere title to a list. \n",
    "This list will contain one dictionary for each hemisphere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup as bs\n",
    "from splinter import Browser\n",
    "import pandas as pd\n",
    "import time \n",
    "import requests\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After a Reset, Curiosity Is Operating Normally\n",
      "NASA's Mars rover Curiosity is in good health but takes a short break while engineers diagnose why it reset its computer. \n"
     ]
    }
   ],
   "source": [
    "#1 Nasa news *** USING BROWSER = SPLINTER ***\n",
    "browser = Browser('chrome')\n",
    "url = \"https://mars.nasa.gov/news/\"\n",
    "#go to url\n",
    "browser.visit(url)\n",
    "# HTML object\n",
    "html = browser.html\n",
    "# Parse HTML with BeautifulSoup\n",
    "soup = bs(html, \"html.parser\")\n",
    "# Collect News Title and Paragraph\n",
    "news_title = soup.find(\"div\", class_ = \"content_title\").text.strip()\n",
    "print(news_title)\n",
    "news_p = soup.find('div', class_=\"article_teaser_body\").text\n",
    "print(news_p)\n",
    "# Close the browser after scraping\n",
    "browser.quit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.jpl.nasa.gov//spaceimages/images/largesize/PIA16883_hires.jpg\n"
     ]
    }
   ],
   "source": [
    "#2- JPL Mars Space Images - Featured Image *** USING BROWSER = SPLINTER ***\n",
    "browser = Browser('chrome')\n",
    "jpl_url = \"https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars)\"\n",
    "#go to url\n",
    "browser.visit(jpl_url)\n",
    "#navigate to link\n",
    "browser.click_link_by_partial_text('FULL IMAGE')\n",
    "time.sleep(2)\n",
    "browser.click_link_by_partial_text('more info')\n",
    "image_html = browser.html\n",
    "jpl_soup = bs(image_html, \"html.parser\")\n",
    "\n",
    "image_path = jpl_soup.find('figure', class_='lede').a['href']\n",
    "featured_image_url = \"https://www.jpl.nasa.gov/\" + image_path\n",
    "print(featured_image_url)\n",
    "# Close the browser after scraping\n",
    "browser.quit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Curiosity is again operating normally following a boot problem first experienced last Friday. Look for more Gale Crater weather conditions soon.\n",
      "https://www.jpl.nasa.gov/news/news.php?feature=7339 …pic.twitter.com/gFMfXyeWDa\n"
     ]
    }
   ],
   "source": [
    "#3- Mars Weather *** USING BROWSER = SPLINTER ***\n",
    "browser = Browser('chrome')\n",
    "url = 'https://twitter.com/marswxreport?lang=en'\n",
    "browser.visit(url)\n",
    "html = browser.html\n",
    "weather_soup = bs(html, 'html.parser')\n",
    "mars_weather = weather_soup.find(\"p\", class_ = \"TweetTextSize TweetTextSize--normal js-tweet-text tweet-text\").text.strip()\n",
    "print(mars_weather)\n",
    "# Close the browser after scraping\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                      0                              1\n",
      "0  Equatorial Diameter:                       6,792 km\n",
      "1       Polar Diameter:                       6,752 km\n",
      "2                 Mass:  6.42 x 10^23 kg (10.7% Earth)\n",
      "3                Moons:            2 (Phobos & Deimos)\n",
      "4       Orbit Distance:       227,943,824 km (1.52 AU)\n",
      "5         Orbit Period:           687 days (1.9 years)\n",
      "6  Surface Temperature:                  -153 to 20 °C\n",
      "7         First Record:              2nd millennium BC\n",
      "8          Recorded By:           Egyptian astronomers]\n"
     ]
    }
   ],
   "source": [
    "#4 Mars Facts\n",
    "#Using Pandas to scrape the table (read_html)\n",
    "url = \"http://space-facts.com/mars/\"\n",
    "table = pd.read_html(url)\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Equatorial Diameter:</td>\n",
       "      <td>6,792 km</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Polar Diameter:</td>\n",
       "      <td>6,752 km</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mass:</td>\n",
       "      <td>6.42 x 10^23 kg (10.7% Earth)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Moons:</td>\n",
       "      <td>2 (Phobos &amp; Deimos)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Orbit Distance:</td>\n",
       "      <td>227,943,824 km (1.52 AU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Orbit Period:</td>\n",
       "      <td>687 days (1.9 years)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Surface Temperature:</td>\n",
       "      <td>-153 to 20 °C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>First Record:</td>\n",
       "      <td>2nd millennium BC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Recorded By:</td>\n",
       "      <td>Egyptian astronomers</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      \n",
       "0  Equatorial Diameter:                       6,792 km\n",
       "1       Polar Diameter:                       6,752 km\n",
       "2                 Mass:  6.42 x 10^23 kg (10.7% Earth)\n",
       "3                Moons:            2 (Phobos & Deimos)\n",
       "4       Orbit Distance:       227,943,824 km (1.52 AU)\n",
       "5         Orbit Period:           687 days (1.9 years)\n",
       "6  Surface Temperature:                  -153 to 20 °C\n",
       "7         First Record:              2nd millennium BC\n",
       "8          Recorded By:           Egyptian astronomers"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Putting the info into Pandas DataFrame\n",
    "mars_facts = table[0]\n",
    "mars_facts.columns = [\"\", \"\"]\n",
    "mars_facts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'<table border=\"1\" class=\"dataframe\">  <thead>    <tr style=\"text-align: right;\">      <th></th>      <th></th>      <th></th>    </tr>  </thead>  <tbody>    <tr>      <th>0</th>      <td>Equatorial Diameter:</td>      <td>6,792 km</td>    </tr>    <tr>      <th>1</th>      <td>Polar Diameter:</td>      <td>6,752 km</td>    </tr>    <tr>      <th>2</th>      <td>Mass:</td>      <td>6.42 x 10^23 kg (10.7% Earth)</td>    </tr>    <tr>      <th>3</th>      <td>Moons:</td>      <td>2 (Phobos &amp; Deimos)</td>    </tr>    <tr>      <th>4</th>      <td>Orbit Distance:</td>      <td>227,943,824 km (1.52 AU)</td>    </tr>    <tr>      <th>5</th>      <td>Orbit Period:</td>      <td>687 days (1.9 years)</td>    </tr>    <tr>      <th>6</th>      <td>Surface Temperature:</td>      <td>-153 to 20 \\xb0C</td>    </tr>    <tr>      <th>7</th>      <td>First Record:</td>      <td>2nd millennium BC</td>    </tr>    <tr>      <th>8</th>      <td>Recorded By:</td>      <td>Egyptian astronomers</td>    </tr>  </tbody></table>'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Using Pandas to convert the data to HTML (to_html)\n",
    "facts_table = mars_facts.to_html()\n",
    "mars_facts_table = facts_table.replace(\"\\n\", \"\")\n",
    "mars_facts_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#5-Mars Hemispheres\n",
    "#create list/dics\n",
    "hemisphere_img_urls = []\n",
    "hemisphere_dicts = {\"title\": [] , \"img_url\": []}\n",
    "# url\n",
    "url = \"https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars\"\n",
    "browser = Browser(\"chrome\")\n",
    "browser.visit(url)\n",
    "home_page = browser.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cerberus Hemisphere Enhanced\n",
      "Cerberus Hemisphere\n",
      "http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/cerberus_enhanced.tif/full.jpg\n",
      "Schiaparelli Hemisphere Enhanced\n",
      "Schiaparelli Hemisphere\n",
      "http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/schiaparelli_enhanced.tif/full.jpg\n",
      "Syrtis Major Hemisphere Enhanced\n",
      "Syrtis Major Hemisphere\n",
      "http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/syrtis_major_enhanced.tif/full.jpg\n",
      "Valles Marineris Hemisphere Enhanced\n",
      "Valles Marineris Hemisphere\n",
      "http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/valles_marineris_enhanced.tif/full.jpg\n"
     ]
    }
   ],
   "source": [
    "#HTML & Parsing\n",
    "hemispheres_soup = bs(home_page, \"html.parser\")\n",
    "results = hemispheres_soup.find_all(\"h3\")\n",
    "# Use loop \n",
    "for result in results:\n",
    "    title = result.text\n",
    "    print(title)\n",
    "    #title without word \"Enhanced\"\n",
    "    title = title[:-9]\n",
    "    print(title)\n",
    "    browser.click_link_by_partial_text(title)\n",
    "    time.sleep(1)\n",
    "    img_url = browser.find_link_by_partial_href(\"download\")[\"href\"]\n",
    "    print(img_url)\n",
    "    hemisphere_dicts = {\"title\": title, \"img_url\": img_url}\n",
    "    hemisphere_img_urls.append(hemisphere_dicts)\n",
    "    time.sleep(1)\n",
    "    browser.visit(url)\n",
    "# Close the browser after scraping\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'img_url': u'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/cerberus_enhanced.tif/full.jpg',\n",
      "  'title': u'Cerberus Hemisphere'},\n",
      " {'img_url': u'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/schiaparelli_enhanced.tif/full.jpg',\n",
      "  'title': u'Schiaparelli Hemisphere'},\n",
      " {'img_url': u'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/syrtis_major_enhanced.tif/full.jpg',\n",
      "  'title': u'Syrtis Major Hemisphere'},\n",
      " {'img_url': u'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/valles_marineris_enhanced.tif/full.jpg',\n",
      "  'title': u'Valles Marineris Hemisphere'}]\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint(hemisphere_img_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2 - MongoDB and Flask Application\n",
    "Use MongoDB with Flask templating to create a new HTML page that displays all of the information that was scraped from the URLs above.\n",
    "\n",
    "1- Start by converting your Jupyter notebook into a Python script called scrape_mars.py with a function called scrape that will execute all of your scraping code from above and return one Python dictionary containing all of the scraped data.\n",
    "\n",
    "2- Next, create a route called /scrape that will import your scrape_mars.py script and call your scrape function.\n",
    "\n",
    "3- Store the return value in Mongo as a Python dictionary.\n",
    "Create a root route / that will query your Mongo database and pass the mars data into an HTML template to display the data.\n",
    "\n",
    "4- Create a template HTML file called index.html that will take the mars data dictionary and display all of the data in the appropriate HTML elements. Use the following as a guide for what the final product should look like, but feel free to create your own design."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}