ImageExtracting_bot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping Instagram with Selenium - UPDATED!\n",
    "\n",
    "### AUTOMATING IMAGE EXTRACTION\n",
    "\n",
    "This Notebook is an <b>upgraded</b> version of the original Web Scraping Instagram with Selenium notebook.\n",
    "<br>\n",
    "This code is asjusted to fit a wider variety of users and has solutions for the issues that arrised in the comment section on YouTube/ issues section on Github.\n",
    "<br>\n",
    "Also, this code is extracting the <b>full size images</b> and not the <b>thumbnails</b>, and it's a <b>100% automated</b>!\n",
    "\n",
    "Please let me know if you have any other problems that you haven't found a solution for in the comment section of the Youtube tutorial:\n",
    "<br>\n",
    "https://youtu.be/iJGvYBH9mcY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#imports here\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.wait import WebDriverWait\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download ChromeDriver\n",
    "Now we need to download latest stable release of ChromeDriver from:\n",
    "<br>\n",
    "https://chromedriver.chromium.org/\n",
    "<br>\n",
    "## Log In to Your Instagram Account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#specify the path to chromedriver.exe (download and save on your computer)\n",
    "driver = webdriver.Firefox()\n",
    "\n",
    "#open the webpage\n",
    "driver.get(\"http://www.instagram.com\")\n",
    "\n",
    "#target username\n",
    "username = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, \"input[name='username']\")))\n",
    "password = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, \"input[name='password']\")))\n",
    "\n",
    "#enter username and password\n",
    "username.clear()\n",
    "username.send_keys(\"my_username\")\n",
    "password.clear()\n",
    "password.send_keys(\"my_password\")\n",
    "\n",
    "#target the login button and click it\n",
    "button = WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, \"button[type='submit']\"))).click()\n",
    "\n",
    "#We are logged in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handle Alerts\n",
    "\n",
    "you might only get a single alert, or you might get 2 of them\n",
    "<br>\n",
    "please adjust the cell below accordingly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(5)\n",
    "alert = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH, '//button[contains(text(), \"Not Now\")]'))).click()\n",
    "#alert2 = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH, '//button[contains(text(), \"Not Now\")]'))).click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search for a certain hashtag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#target the search input field\n",
    "searchbox = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, \"//input[@placeholder='Search']\")))\n",
    "searchbox.clear()\n",
    "\n",
    "#search for the hashtag cat\n",
    "keyword = \"#cat\"\n",
    "searchbox.send_keys(keyword)\n",
    " \n",
    "#FIXING THE DOUBLE ENTER\n",
    "time.sleep(5) # Wait for 5 seconds\n",
    "my_link = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, \"//a[contains(@href, '/\" + keyword[1:] + \"/')]\")))\n",
    "my_link.click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scroll Down\n",
    "<br>\n",
    "Increase n_scrolls to select more photos (depending on screen resolution)\n",
    "<br>\n",
    "<b>Example:</b>\n",
    "<br>\n",
    "<ul>\n",
    "    <li>2 scrolls cover approx. 35 photos</li>\n",
    "    <li>3 scrolls cover approx. 45 photos</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#scroll down 2 times\n",
    "#increase the range to sroll more\n",
    "n_scrolls = 2\n",
    "for j in range(0, n_scrolls):\n",
    "    driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 45 links to images\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://www.instagram.com/p/CLS2iQYMMQl/',\n",
       " 'https://www.instagram.com/p/CLTZ6wssjRb/',\n",
       " 'https://www.instagram.com/p/CLTF0NID96i/',\n",
       " 'https://www.instagram.com/p/CLTGXXipfmn/',\n",
       " 'https://www.instagram.com/p/CLTR-boJOYx/']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#target all the link elements on the page\n",
    "anchors = driver.find_elements_by_tag_name('a')\n",
    "anchors = [a.get_attribute('href') for a in anchors]\n",
    "#narrow down all links to image links only\n",
    "anchors = [a for a in anchors if str(a).startswith(\"https://www.instagram.com/p/\")]\n",
    "\n",
    "print('Found ' + str(len(anchors)) + ' links to images')\n",
    "anchors[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://instagram.fcxh2-1.fna.fbcdn.net/v/t51.2885-15/e35/150845044_422240689052825_5373860191640002462_n.jpg?_nc_ht=instagram.fcxh2-1.fna.fbcdn.net&_nc_cat=105&_nc_ohc=Rglzept-_sYAX8otSon&tp=1&oh=8413ecf31e264fa9e510f73d4e68daf9&oe=6053936C',\n",
       " 'https://instagram.fcxh2-1.fna.fbcdn.net/v/t51.2885-15/e35/s1080x1080/149845587_456976818986058_3313267170358432373_n.jpg?_nc_ht=instagram.fcxh2-1.fna.fbcdn.net&_nc_cat=106&_nc_ohc=MwBvUuk3lW0AX8mVE9L&tp=1&oh=7c666dd5c7ac66b93765362eb8b74241&oe=6054B236',\n",
       " 'https://instagram.fcxh2-1.fna.fbcdn.net/v/t51.2885-19/s150x150/15337291_176285289506077_2320742856353906688_a.jpg?_nc_ht=instagram.fcxh2-1.fna.fbcdn.net&_nc_ohc=x_WkwQRWQPwAX-wgXTI&tp=1&oh=15f45ac2f2df34926ffaffbeaa5b3f34&oe=60533914',\n",
       " 'https://instagram.fcxh2-1.fna.fbcdn.net/v/t51.2885-15/fr/e15/s1080x1080/149426673_2944148305872024_5738325172882694686_n.jpg?_nc_ht=instagram.fcxh2-1.fna.fbcdn.net&_nc_cat=100&_nc_ohc=yqLRfNFvfUgAX9oTXGK&tp=1&oh=e7048954fe837771452d8830f1dd43d1&oe=605322EC',\n",
       " 'https://instagram.fcxh2-1.fna.fbcdn.net/v/t51.2885-15/e35/s1080x1080/149730534_467521324608969_2970791019083350946_n.jpg?_nc_ht=instagram.fcxh2-1.fna.fbcdn.net&_nc_cat=109&_nc_ohc=LX2bwbgOb1MAX_3H9Zo&tp=1&oh=c760fa0fd4f5c451e687286ad6135ce6&oe=6054C970']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images = []\n",
    "\n",
    "#follow each image link and extract only image at index=1\n",
    "for a in anchors:\n",
    "    driver.get(a)\n",
    "    time.sleep(5)\n",
    "    img = driver.find_elements_by_tag_name('img')\n",
    "    img = [i.get_attribute('src') for i in img]\n",
    "    images.append(img[1])\n",
    "    \n",
    "images[:5]    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save images to computer\n",
    "\n",
    "First we'll create a new folder for our images somewhere on our computer.\n",
    "<br>\n",
    "Then, we'll save all the images there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\goaim\\\\cats'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import wget\n",
    "\n",
    "path = os.getcwd()\n",
    "path = os.path.join(path, keyword[1:] + \"s\")\n",
    "\n",
    "#create the directory\n",
    "os.mkdir(path)\n",
    "\n",
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100% [............................................................................] 212998 / 212998"
     ]
    }
   ],
   "source": [
    "#download images\n",
    "counter = 0\n",
    "for image in images:\n",
    "    save_as = os.path.join(path, keyword[1:] + str(counter) + '.jpg')\n",
    "    wget.download(image, save_as)\n",
    "    counter += 1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}