IntroToDataScience.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "IntroToDataScience.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/womenwhocodedc/python-community/blob/master/IntroToDataScience.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1o5gj4ADOoNh",
        "colab_type": "text"
      },
      "source": [
        "# Handling Dirty and Missing Data Using Pandas\n",
        "\n",
        "Presented by\n",
        "\n",
        "Saba Ali and Marissa Weiner\n",
        "\n",
        "Authors:\n",
        "\n",
        "Sian Lewis, Booz Allen Hamilton, sian.lewis@gmail.com\n",
        "\n",
        "Valeria Rozenbaum, Thomson Reuters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXOW_3KjyxS4",
        "colab_type": "text"
      },
      "source": [
        "#### NOTE: OPEN IN PLAYGROUND MODE IF YOU CAN\n",
        "\n",
        "# ABOUT THE DATASET\n",
        "\n",
        "**The dataset we're working with today comes from the  [FlourishOA github repo](https://github.com/FlourishOA/Data)**\n",
        "\n",
        "\n",
        "**[FlourishOA](http://flourishoa.org/) was created by University of Washington's Information School Data Science and Analytics Lab to allow for the discovery of reputable open access publications . \n",
        "The data was compiled from a variety of sources, including researchers, web scraping, and the publishers themselves.**\n",
        "\n",
        "**Our dataset consists the following columns:**\n",
        "\n",
        "* **issn** international standardized code which identifies journals, magazines, and publications of both print and electronica media\n",
        "* **journal_name** name of the journal\n",
        "* **pub_name** name of the publisher\n",
        "* **is_hybrid**   definition unknown\n",
        "* **category**  subject classification of the journal\n",
        "* **url** web link to the publication\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1XAgL7AUysWU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## First things first, let's import the packages we'll be using\n",
        "\n",
        "import pandas as pd # Pandas library for data manipulation \n",
        "import numpy as np  # Numpy library for scientific computation \n",
        "import io #for allowing pandas.read_csv to read in the file in google colab\n",
        "import requests # Allows requesting url\n",
        "\n",
        "####################################### \n",
        "#######################################\n",
        "\n",
        "url = 'https://raw.githubusercontent.com/FlourishOA/Data/master/estimated-article-influence-scores-2015.csv'\n",
        "df = pd.read_csv(url)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWq2FMINPKqm",
        "colab_type": "text"
      },
      "source": [
        "# Part 1: Cleaning Dirty Data \n",
        "\n",
        " **Python References and Documentation**\n",
        "*  [Python Encodings](https://docs.python.org/3.7/library/codecs.html#encodings-and-unicode) \n",
        "*  [Pandas Read CSV](https://docs.python.org/2.4/lib/standard-encodings.html)\n",
        "* [Regular Expressions](https://docs.python.org/3.4/howto/regex.html)\n",
        "* [Regular Expressions with Pandas](https://www.dataquest.io/blog/regular-expressions-data-scientists/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e7i58AyyylBp",
        "colab_type": "text"
      },
      "source": [
        "## 1.a: Encoding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g-nQIl589qBE",
        "colab_type": "text"
      },
      "source": [
        "Character encodings allow your computer to interpret binary bytes (ex:111010101101) as real characters. This is done by mapping human readable characters to byte strings.\n",
        "\n",
        "While there are many encodings, UTF-8 is one of the most commonly used and is often the default for Python.\n",
        "\n",
        "Because of this, our encoding problems don't become problems until we have non UTF-8 data.  \n",
        "\n",
        "**General Terms**\n",
        "- Decoding - To convert a string of bytes to a unicode string aka human readable.\n",
        "- Encoding - To represent a unicode string as a string of bytes.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJ2kWewWCLJx",
        "colab_type": "code",
        "outputId": "adc55e6b-0572-4fa8-be67-6d58efeaa95c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 867
        }
      },
      "source": [
        "# We don't have to specify the encoding when reading in a csv using pandas and most of the time there aren't any issues because the data is likely utf8 encoded\n",
        "# Spoiler alert: this data is not\n",
        "\n",
        "this_will_fail = pd.read_csv('https://raw.githubusercontent.com/FlourishOA/Data/master/api_journal11-13-17.csv')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "UnicodeDecodeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mUnicodeDecodeError\u001b[0m                        Traceback (most recent call last)",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_tokens\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_with_dtype\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._string_convert\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers._string_box_utf8\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0xed in position 17: invalid continuation byte",
            "\nDuring handling of the above exception, another exception occurred:\n",
            "\u001b[0;31mUnicodeDecodeError\u001b[0m                        Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-15-ccc38eacbf08>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mthis_will_fail\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'https://raw.githubusercontent.com/FlourishOA/Data/master/api_journal11-13-17.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    700\u001b[0m                     skip_blank_lines=skip_blank_lines)\n\u001b[1;32m    701\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 702\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    703\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    704\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    433\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 435\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    436\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    437\u001b[0m         \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m   1137\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1138\u001b[0m         \u001b[0mnrows\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_validate_integer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'nrows'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1139\u001b[0;31m         \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1140\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1141\u001b[0m         \u001b[0;31m# May alter columns / col_dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m   1993\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1994\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1995\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1996\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1997\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_first_chunk\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.read\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_low_memory\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_rows\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_column_data\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_tokens\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_with_dtype\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._string_convert\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers._string_box_utf8\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0xed in position 17: invalid continuation byte"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r3n_2u7Cf_xR",
        "colab_type": "code",
        "outputId": "b6fc8e04-264d-4f35-b415-c473e49d2b7c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#Lets try reading in the actual csv file we uploaded at the top and specify decoding to utf8\n",
        "s = requests.get(url).content\n",
        "utf8_decode = pd.read_csv(io.StringIO(s.decode('utf-8')))\n",
        "utf8_jn_list = utf8_decode['journal_name'].unique().tolist()\n",
        "\n",
        "\n",
        "utf8_jn_list[:500]  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['3d research',\n",
              " 'aaps pharmscitech',\n",
              " 'abstract and applied analysis',\n",
              " 'academic psychiatry',\n",
              " 'academic questions',\n",
              " 'accreditation and quality assurance',\n",
              " 'acoustics australia',\n",
              " 'acrocephalus',\n",
              " 'acta adriatica',\n",
              " 'acta agriculturae slovenica',\n",
              " 'acta amazonica',\n",
              " 'acta applicandae mathematicae',\n",
              " 'acta biochimica polonica',\n",
              " 'acta bioethica',\n",
              " 'acta biologica cracoviensia series botanica',\n",
              " 'acta biotheoretica',\n",
              " 'acta botanica brasilica',\n",
              " 'acta botanica croatica',\n",
              " 'acta chimica sinica',\n",
              " 'acta chimica slovenica',\n",
              " 'acta cirurgica brasileira',\n",
              " 'acta clinica croatica',\n",
              " 'acta diabetologica',\n",
              " 'acta endoscopica',\n",
              " 'acta ethologica',\n",
              " 'acta geophysica',\n",
              " 'acta geotechnica',\n",
              " 'acta herpetologica',\n",
              " 'acta histochemica et cytochemica',\n",
              " 'acta ichthyologica et piscatoria',\n",
              " 'acta informatica',\n",
              " 'acta limnologica brasiliensia',\n",
              " 'acta mechanica',\n",
              " 'acta mechanica sinica',\n",
              " 'acta medica okayama',\n",
              " 'acta medica portuguesa',\n",
              " 'acta montanistica slovaca',\n",
              " 'acta neurobiologiae experimentalis',\n",
              " 'acta neurochirurgica',\n",
              " 'acta neurologica belgica',\n",
              " 'acta neuropathologica',\n",
              " 'acta orthopaedica',\n",
              " 'acta orthopaedica et traumatologica turcica',\n",
              " 'acta ortopedica brasileira',\n",
              " 'acta palaeontologica polonica',\n",
              " 'acta paulista de enfermagem',\n",
              " 'acta pharmaceutica',\n",
              " 'acta physica polonica a',\n",
              " 'acta physiologiae plantarum',\n",
              " 'acta polytechnica hungarica',\n",
              " 'acta protozoologica',\n",
              " 'acta reumatologica portuguesa',\n",
              " 'acta societatis botanicorum poloniae',\n",
              " 'acta veterinaria brno',\n",
              " 'acta veterinaria scandinavica',\n",
              " 'acta zoologica academiae scientiarum hungaricae',\n",
              " 'active and passive electronic components',\n",
              " 'adansonia',\n",
              " 'addiction science & clinical practice',\n",
              " 'adhd attention deficit and hyperactivity disorders',\n",
              " 'advanced science letters',\n",
              " 'advances in acoustics and vibration',\n",
              " 'advances in applied clifford algebras',\n",
              " 'advances in artificial intelligence',\n",
              " 'advances in artificial neural systems',\n",
              " 'advances in astronomy',\n",
              " 'advances in atmospheric sciences',\n",
              " 'advances in bioinformatics',\n",
              " 'advances in civil engineering',\n",
              " 'advances in cognitive psychology',\n",
              " 'advances in computational mathematics',\n",
              " 'advances in condensed matter physics',\n",
              " 'advances in decision sciences',\n",
              " 'advances in difference equations',\n",
              " 'advances in electrical and computer engineering',\n",
              " 'advances in fuzzy systems',\n",
              " 'advances in geosciences',\n",
              " 'advances in health sciences education',\n",
              " 'advances in high energy physics',\n",
              " 'advances in materials science and engineering',\n",
              " 'advances in mathematical physics',\n",
              " 'advances in mechanical engineering',\n",
              " 'advances in medical sciences',\n",
              " 'advances in meteorology',\n",
              " 'advances in numerical analysis',\n",
              " 'advances in operations research',\n",
              " 'advances in optical technologies',\n",
              " 'advances in optoelectronics',\n",
              " 'advances in pharmacological sciences',\n",
              " 'advances in physical chemistry',\n",
              " 'advances in physiology education',\n",
              " 'advances in rehabilitation',\n",
              " 'advances in science and research',\n",
              " 'advances in software engineering',\n",
              " 'advances in therapy',\n",
              " 'advances in tribology',\n",
              " 'advances in urology',\n",
              " 'aequationes mathematicae',\n",
              " 'aerobiologia',\n",
              " 'aesthetic plastic surgery',\n",
              " 'africa spectrum',\n",
              " 'african archaeological review',\n",
              " 'african invertebrates',\n",
              " 'african journal of agricultural research',\n",
              " 'african journal of biotechnology',\n",
              " 'african journal of business management',\n",
              " 'african journal of microbiology research',\n",
              " 'african journal of paediatric surgery',\n",
              " 'african journal of pharmacy and pharmacology',\n",
              " 'african journal of primary health care & family medicine',\n",
              " 'age',\n",
              " 'ageing international',\n",
              " 'aging cell',\n",
              " 'aging clinical and experimental research',\n",
              " 'agricultural and food science',\n",
              " 'agriculture and human values',\n",
              " 'agrociencia',\n",
              " 'agroforestry systems',\n",
              " 'agronomy for sustainable development',\n",
              " 'ai & society',\n",
              " 'aids and behavior',\n",
              " 'aids research and therapy',\n",
              " 'aids research and treatment',\n",
              " 'aip advances',\n",
              " 'alcohol',\n",
              " 'alcohol research & health',\n",
              " 'algebra universalis',\n",
              " 'algebras and representation theory',\n",
              " 'algorithmica',\n",
              " 'algorithms',\n",
              " 'algorithms for molecular biology',\n",
              " 'allergology international',\n",
              " 'alpine botany',\n",
              " 'amb express',\n",
              " 'american journal of animal and veterinary sciences',\n",
              " 'american journal of applied sciences',\n",
              " 'american journal of cancer research',\n",
              " 'american journal of cardiovascular drugs',\n",
              " 'american journal of clinical dermatology',\n",
              " 'american journal of criminal justice',\n",
              " 'american journal of dance therapy',\n",
              " 'american journal of engineering and applied sciences',\n",
              " 'american journal of potato research',\n",
              " 'american journal of translational research',\n",
              " 'amfiteatru economic',\n",
              " 'amino acids',\n",
              " 'anaerobe',\n",
              " 'anais brasileiros de dermatologia',\n",
              " 'anales de psicologia',\n",
              " 'anales del jardin botanico de madrid',\n",
              " 'analog integrated circuits and signal processing',\n",
              " 'analytical and bioanalytical chemistry',\n",
              " 'analytical cellular pathology',\n",
              " 'analytical sciences',\n",
              " 'anatolian journal of cardiology',\n",
              " 'anatomical science international',\n",
              " 'andean geology',\n",
              " 'anemia',\n",
              " 'anesthesiology research and practice',\n",
              " 'angiogenesis',\n",
              " 'animal biodiversity and conservation',\n",
              " 'animal cognition',\n",
              " 'animal science papers and reports',\n",
              " 'annales geophysicae',\n",
              " 'annali dell istituto superiore di sanita',\n",
              " 'annals academy of medicine singapore',\n",
              " 'annals of agricultural and environmental medicine',\n",
              " 'annals of behavioral medicine',\n",
              " 'annals of biomedical engineering',\n",
              " 'annals of cardiac anaesthesia',\n",
              " 'annals of clinical microbiology and antimicrobials',\n",
              " 'annals of combinatorics',\n",
              " 'annals of dyslexia',\n",
              " 'annals of family medicine',\n",
              " 'annals of finance',\n",
              " 'annals of forest research',\n",
              " 'annals of forest science',\n",
              " 'annals of general psychiatry',\n",
              " 'annals of geophysics',\n",
              " 'annals of glaciology',\n",
              " 'annals of global analysis and geometry',\n",
              " 'annals of hematology',\n",
              " 'annals of indian academy of neurology',\n",
              " 'annals of intensive care',\n",
              " 'annals of mathematics and artificial intelligence',\n",
              " 'annals of microbiology',\n",
              " 'annals of nuclear medicine',\n",
              " 'annals of operations research',\n",
              " 'annals of pediatric cardiology',\n",
              " 'annals of saudi medicine',\n",
              " 'annals of solid and structural mechanics',\n",
              " 'annals of surgical innovation and research',\n",
              " 'annals of surgical oncology',\n",
              " 'annals of thoracic and cardiovascular surgery',\n",
              " 'annals of thoracic medicine',\n",
              " 'annals of tropical medicine and public health',\n",
              " 'anthropological review',\n",
              " 'anthropological science',\n",
              " 'anuario colombiano de historia social y de la cultura',\n",
              " 'anuario de estudios americanos',\n",
              " 'anuario de estudios medievales',\n",
              " 'aob plants',\n",
              " 'apidologie',\n",
              " 'apl materials',\n",
              " 'apoptosis',\n",
              " 'appetite',\n",
              " 'applicable analysis and discrete mathematics',\n",
              " 'applied and environmental soil science',\n",
              " 'applied biochemistry and biotechnology',\n",
              " 'applied bionics and biomechanics',\n",
              " 'applied categorical structures',\n",
              " 'applied composite materials',\n",
              " 'applied ecology and environmental research',\n",
              " 'applied entomology and zoology',\n",
              " 'applied geomatics',\n",
              " 'applied health economics and health policy',\n",
              " 'applied informatics',\n",
              " 'applied intelligence',\n",
              " 'applied magnetic resonance',\n",
              " 'applied microbiology and biotechnology',\n",
              " 'applied physics a',\n",
              " 'applied physics b',\n",
              " 'applied psychophysiology and biofeedback',\n",
              " 'applied research in quality of life',\n",
              " 'applied spatial analysis and policy',\n",
              " 'aquaculture',\n",
              " 'aquaculture environment interactions',\n",
              " 'aquaculture international',\n",
              " 'aquatic biology',\n",
              " 'aquatic ecology',\n",
              " 'aquatic geochemistry',\n",
              " 'aquatic invasions',\n",
              " 'aquatic sciences',\n",
              " 'aquichan',\n",
              " 'arabian journal for science and engineering',\n",
              " 'arabian journal of chemistry',\n",
              " 'arabian journal of geosciences',\n",
              " 'archaeological and anthropological sciences',\n",
              " 'archiv der mathematik',\n",
              " 'archival science',\n",
              " 'archive for history of exact sciences',\n",
              " 'archive for mathematical logic',\n",
              " 'archive for rational mechanics and analysis',\n",
              " 'archive of applied mechanics',\n",
              " 'archives of acoustics',\n",
              " 'archives of biological sciences',\n",
              " 'archives of computational methods in engineering',\n",
              " 'archives of dermatological research',\n",
              " 'archives of environmental contamination and toxicology',\n",
              " 'archives of environmental protection',\n",
              " 'archives of gynecology and obstetrics',\n",
              " 'archives of histology and cytology',\n",
              " 'archives of mechanics',\n",
              " 'archives of medical science',\n",
              " 'archives of metallurgy and materials',\n",
              " 'archives of microbiology',\n",
              " 'archives of orthopaedic and trauma surgery',\n",
              " 'archives of osteoporosis',\n",
              " 'archives of pharmacal research',\n",
              " 'archives of polish fisheries',\n",
              " 'archives of scientific psychology',\n",
              " 'archives of sexual behavior',\n",
              " 'archives of toxicology',\n",
              " 'archives of virology',\n",
              " 'archivos de medicina',\n",
              " 'archivos de zootecnia',\n",
              " 'archivum immunologiae et therapiae experimentalis',\n",
              " 'argumentation',\n",
              " 'arhiv za higijenu rada i toksikologiju',\n",
              " 'arkivoc',\n",
              " 'arquivo brasileiro de medicina veterinaria e zootecnia',\n",
              " 'arquivos brasileiros de cardiologia',\n",
              " 'arquivos brasileiros de oftalmologia',\n",
              " 'arquivos de gastroenterologia',\n",
              " 'arthritis',\n",
              " 'arthritis research & therapy',\n",
              " 'arthroskopie',\n",
              " 'artificial intelligence and law',\n",
              " 'artificial intelligence review',\n",
              " 'artificial life and robotics',\n",
              " 'asia europe journal',\n",
              " 'asia pacific education review',\n",
              " 'asia pacific family medicine',\n",
              " 'asia pacific journal of management',\n",
              " 'asian journal of algebra',\n",
              " 'asian journal of andrology',\n",
              " 'asian journal of applied sciences',\n",
              " 'asian journal of chemistry',\n",
              " 'asian journal of clinical nutrition',\n",
              " 'asian journal of criminology',\n",
              " 'asian journal of crop science',\n",
              " 'asian journal of epidemiology',\n",
              " 'asian journal of mathematics & statistics',\n",
              " 'asian journal of pharmaceutics',\n",
              " 'asian journal of plant sciences',\n",
              " 'asian journal of scientific research',\n",
              " 'asian journal of surgery',\n",
              " 'asian journal of transfusion science',\n",
              " 'asian pacific journal of cancer prevention',\n",
              " 'asn neuro',\n",
              " 'asta advances in statistical analysis',\n",
              " 'astrophysics and space science',\n",
              " 'atencion primaria',\n",
              " 'atherosclerosis',\n",
              " 'atlantic economic journal',\n",
              " 'atmosfera',\n",
              " 'atmosphere',\n",
              " 'atmospheric chemistry and physics',\n",
              " 'atmospheric measurement techniques',\n",
              " 'atmospheric science letters',\n",
              " 'audiology',\n",
              " 'australasian journal of educational technology',\n",
              " 'australasian journal of information systems',\n",
              " 'australasian medical journal',\n",
              " 'australasian physical & engineering sciences in medicine',\n",
              " 'australasian plant pathology',\n",
              " 'australia and new zealand health policy',\n",
              " 'australian journal of crop science',\n",
              " 'austrian journal of earth sciences',\n",
              " 'autex research journal',\n",
              " 'autism research and treatment',\n",
              " 'autoimmune diseases',\n",
              " 'automated experimentation',\n",
              " 'automatica',\n",
              " 'autonomous robots',\n",
              " 'avian conservation and ecology',\n",
              " 'axiomathes',\n",
              " 'balkan journal of geometry and its applications',\n",
              " 'balkan journal of medical genetics',\n",
              " 'balkan medical journal',\n",
              " 'baltic journal of law & politics',\n",
              " 'baltica',\n",
              " 'bangladesh journal of pharmacology',\n",
              " 'bangladesh journal of plant taxonomy',\n",
              " 'basic research in cardiology',\n",
              " 'bayesian analysis',\n",
              " 'behavior genetics',\n",
              " 'behavior research methods',\n",
              " 'behavioral and brain functions',\n",
              " 'behavioral ecology and sociobiology',\n",
              " 'behavioural neurology',\n",
              " 'beilstein journal of nanotechnology',\n",
              " 'beilstein journal of organic chemistry',\n",
              " 'biocell',\n",
              " 'biochemia medica',\n",
              " 'biochemical genetics',\n",
              " 'biochemistry research international',\n",
              " 'biochimie',\n",
              " 'biocontrol',\n",
              " 'biocontrol science',\n",
              " 'biodata mining',\n",
              " 'biodegradation',\n",
              " 'biodiversity and conservation',\n",
              " 'biodrugs',\n",
              " 'bioelectrochemistry',\n",
              " 'bioenergy research',\n",
              " 'biogeochemistry',\n",
              " 'biogeosciences',\n",
              " 'biogeosciences discussions',\n",
              " 'biogerontology',\n",
              " 'bioinformatics and biology insights',\n",
              " 'bioinorganic chemistry and applications',\n",
              " 'biointerphases',\n",
              " 'biologia plantarum',\n",
              " 'biological & pharmaceutical bulletin',\n",
              " 'biological cybernetics',\n",
              " 'biological invasions',\n",
              " 'biological letters',\n",
              " 'biological procedures online',\n",
              " 'biological research',\n",
              " 'biological theory',\n",
              " 'biological trace element research',\n",
              " 'biologicals',\n",
              " 'biology and fertility of soils',\n",
              " 'biology direct',\n",
              " 'biology of sex differences',\n",
              " 'biology of sport',\n",
              " 'biomarker insights',\n",
              " 'biomaterials',\n",
              " 'biomechanics and modeling in mechanobiology',\n",
              " 'biomed research international',\n",
              " 'biomedica',\n",
              " 'biomedical engineering online',\n",
              " 'biomedical microdevices',\n",
              " 'biometals',\n",
              " 'biomicrofluidics',\n",
              " 'biomolecular nmr assignments',\n",
              " 'biophysical reviews',\n",
              " 'bioprocess and biosystems engineering',\n",
              " 'biopsychosocial medicine',\n",
              " 'bioresources',\n",
              " 'bioscience horizons',\n",
              " 'bioscience reports',\n",
              " 'bioscience trends',\n",
              " 'biosemiotics',\n",
              " 'biosystems',\n",
              " 'biota neotropica',\n",
              " 'biotechnologie agronomie societe et environnement',\n",
              " 'biotechnology & biotechnological equipment',\n",
              " 'biotechnology for biofuels',\n",
              " 'biotechnology letters',\n",
              " 'biotechnology research international',\n",
              " 'bit numerical mathematics',\n",
              " 'blood cancer journal',\n",
              " 'blood transfusion',\n",
              " 'bmc anesthesiology',\n",
              " 'bmc biochemistry',\n",
              " 'bmc bioinformatics',\n",
              " 'bmc biology',\n",
              " 'bmc biophysics',\n",
              " 'bmc biotechnology',\n",
              " 'bmc blood disorders',\n",
              " 'bmc cancer',\n",
              " 'bmc cardiovascular disorders',\n",
              " 'bmc cell biology',\n",
              " 'bmc chemical biology',\n",
              " 'bmc clinical pathology',\n",
              " 'bmc clinical pharmacology',\n",
              " 'bmc complementary and alternative medicine',\n",
              " 'bmc dermatology',\n",
              " 'bmc developmental biology',\n",
              " 'bmc ecology',\n",
              " 'bmc emergency medicine',\n",
              " 'bmc endocrine disorders',\n",
              " 'bmc evolutionary biology',\n",
              " 'bmc family practice',\n",
              " 'bmc gastroenterology',\n",
              " 'bmc genetics',\n",
              " 'bmc genomics',\n",
              " 'bmc geriatrics',\n",
              " 'bmc health services research',\n",
              " 'bmc immunology',\n",
              " 'bmc infectious diseases',\n",
              " 'bmc international health and human rights',\n",
              " 'bmc medical education',\n",
              " 'bmc medical ethics',\n",
              " 'bmc medical genetics',\n",
              " 'bmc medical genomics',\n",
              " 'bmc medical imaging',\n",
              " 'bmc medical informatics and decision making',\n",
              " 'bmc medical physics',\n",
              " 'bmc medical research methodology',\n",
              " 'bmc medicine',\n",
              " 'bmc microbiology',\n",
              " 'bmc molecular biology',\n",
              " 'bmc musculoskeletal disorders',\n",
              " 'bmc nephrology',\n",
              " 'bmc neurology',\n",
              " 'bmc neuroscience',\n",
              " 'bmc nursing',\n",
              " 'bmc ophthalmology',\n",
              " 'bmc oral health',\n",
              " 'bmc palliative care',\n",
              " 'bmc pediatrics',\n",
              " 'bmc pharmacology',\n",
              " 'bmc physiology',\n",
              " 'bmc plant biology',\n",
              " 'bmc pregnancy and childbirth',\n",
              " 'bmc proceedings',\n",
              " 'bmc psychiatry',\n",
              " 'bmc public health',\n",
              " 'bmc pulmonary medicine',\n",
              " 'bmc research notes',\n",
              " 'bmc structural biology',\n",
              " 'bmc surgery',\n",
              " 'bmc systems biology',\n",
              " 'bmc urology',\n",
              " 'bmc veterinary research',\n",
              " 'bmj open',\n",
              " 'boletin de la sociedad argentina de botanica',\n",
              " 'bone',\n",
              " 'bone marrow research',\n",
              " 'boreal environment research',\n",
              " 'bosnian journal of basic medical sciences',\n",
              " 'botanical studies',\n",
              " 'boundary value problems',\n",
              " 'brachytherapy',\n",
              " 'bragantia',\n",
              " 'brain imaging and behavior',\n",
              " 'brain sciences',\n",
              " 'brain topography',\n",
              " 'brain tumor pathology',\n",
              " 'brazilian archives of biology and technology',\n",
              " 'brazilian journal of biology',\n",
              " 'brazilian journal of botany',\n",
              " 'brazilian journal of cardiovascular surgery',\n",
              " 'brazilian journal of chemical engineering',\n",
              " 'brazilian journal of infectious diseases',\n",
              " 'brazilian journal of medical and biological research',\n",
              " 'brazilian journal of microbiology',\n",
              " 'brazilian journal of oceanography',\n",
              " 'brazilian journal of pharmaceutical sciences',\n",
              " 'brazilian journal of physics',\n",
              " 'brazilian journal of poultry science',\n",
              " 'brazilian journal of veterinary research and animal science',\n",
              " 'brazilian oral research',\n",
              " 'breast cancer',\n",
              " 'breast cancer research',\n",
              " 'breast cancer research and treatment',\n",
              " 'breathe']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7ees_KAmOTU",
        "colab_type": "text"
      },
      "source": [
        "**We are able to read the file in. Sometimes data will have the \"?\" character. If that is the case, we must do a different type of encoding.**\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "**The ? signifies an unknown character and means that there is no utf8 mapping for that particular byte to be translated.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6T51VHqaCLaF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Let's try using a different encoding \n",
        "# Fun fact...kind of: latin1 is an alias for ISO-8859-1 \n",
        "\n",
        "df = pd.read_csv('https://raw.githubusercontent.com/FlourishOA/Data/master/api_journal11-13-17.csv',encoding='ISO-8859-1') \n",
        "jn_list = df['journal_name'].unique().tolist()\n",
        "\n",
        "\n",
        "jn_list[:500]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x7ub1S8emy81",
        "colab_type": "text"
      },
      "source": [
        "#### With the correct encoding, we can read the text from our previous example and see what the unknown characters were\n",
        "\n",
        "\n",
        "\n",
        "*   utf8:\n",
        "\n",
        "Onom?zein : Revista De Ling??stica, Filolog?a Y Traducci?nOnomázein\n",
        "*   ISO-8859-1:        \n",
        "\n",
        "Onomázein : Revista De Lingüística, Filología Y Traducción\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hcoW-qk0wNH8",
        "colab_type": "code",
        "outputId": "e0002b1a-33ee-4d17-a1ac-ca8d1b9c76b4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "## Let's do a quick profile of our dataset \n",
        "\n",
        "def eda(dataframe):\n",
        "    print('MISSING VALUES\\n', dataframe.isnull().sum())\n",
        "    print('\\n DATA TYPES \\n', dataframe.dtypes)\n",
        "    print('\\n DATA SHAPE \\n', dataframe.shape)\n",
        "    print('\\n DATA DESCRIBE \\n', dataframe.describe())\n",
        "    for item in dataframe:\n",
        "        print('\\n UNIQUE VALUE TOTALS \\n',item)\n",
        "        print(dataframe[item].nunique())\n",
        "        \n",
        "eda(df)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MISSING VALUES\n",
            " issn               0\n",
            "journal_name       0\n",
            "pub_name        1470\n",
            "is_hybrid          0\n",
            "category        6331\n",
            "url             6788\n",
            "dtype: int64\n",
            "\n",
            " DATA TYPES \n",
            " issn            object\n",
            "journal_name    object\n",
            "pub_name        object\n",
            "is_hybrid        int64\n",
            "category        object\n",
            "url             object\n",
            "dtype: object\n",
            "\n",
            " DATA SHAPE \n",
            " (13149, 6)\n",
            "\n",
            " DATA DESCRIBE \n",
            "           is_hybrid\n",
            "count  13149.000000\n",
            "mean       0.131949\n",
            "std        0.338448\n",
            "min        0.000000\n",
            "25%        0.000000\n",
            "50%        0.000000\n",
            "75%        0.000000\n",
            "max        1.000000\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " issn\n",
            "13149\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " journal_name\n",
            "11581\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " pub_name\n",
            "3529\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " is_hybrid\n",
            "2\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " category\n",
            "140\n",
            "\n",
            " UNIQUE VALUE TOTALS \n",
            " url\n",
            "6327\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SNA0T8_ldxI",
        "colab_type": "text"
      },
      "source": [
        "## 1.b: Deduplication"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJqVc6Y1lCrb",
        "colab_type": "code",
        "outputId": "06f3822e-87e3-420c-8e45-d3978409fc41",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Lets check if we have any serial number duplication\n",
        "\n",
        "\n",
        "print(any(df['issn'].duplicated())) #any is a method that iterates through strings and returns a boolean value based on the condition. \n",
        "                                    #returning false means that no duplicate values were found in issn"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uROrwUJ4JqrJ",
        "colab_type": "code",
        "outputId": "ad312641-2e7e-47bf-a56b-ede5946b7d5a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#We can also check for duplication in multiple columns using the same logic\n",
        "\n",
        "print(any(df.duplicated(['journal_name', 'pub_name','url']))) "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C6i6mK9sNwfR",
        "colab_type": "code",
        "outputId": "54f19bcb-5f78-4d79-c83a-63f1c3a986da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#Lets see the dupes\n",
        "\n",
        "df[df.duplicated(['journal_name', 'pub_name','url'], keep='first')] #specifying keep=first counts the first occurring value as original and the remaining as duplicates\n",
        "\n",
        "#Looks like we ran into a problem with identifying actual dupe values.. because nulls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>issn</th>\n",
              "      <th>journal_name</th>\n",
              "      <th>pub_name</th>\n",
              "      <th>is_hybrid</th>\n",
              "      <th>category</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>0053-0016</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>133</th>\n",
              "      <td>0053-0017</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>134</th>\n",
              "      <td>0053-0018</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>135</th>\n",
              "      <td>0053-0019</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>136</th>\n",
              "      <td>0053-0020</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>137</th>\n",
              "      <td>0053-0021</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>138</th>\n",
              "      <td>0053-0022</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>139</th>\n",
              "      <td>0053-0023</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>140</th>\n",
              "      <td>0053-0024</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>141</th>\n",
              "      <td>0053-0025</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142</th>\n",
              "      <td>0053-0026</td>\n",
              "      <td>International Journal of Recent Surgical and M...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>435</th>\n",
              "      <td>0719-4986</td>\n",
              "      <td>Biodiversity and Natural History</td>\n",
              "      <td>Centro de Estudios en Biodiversidad</td>\n",
              "      <td>0</td>\n",
              "      <td>Science</td>\n",
              "      <td>http://www.biodiversnathist.com</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>454</th>\n",
              "      <td>0812-1126</td>\n",
              "      <td>Biointerface Research in Applied Chemistry</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>455</th>\n",
              "      <td>0812-1135</td>\n",
              "      <td>Biointerface Research in Applied Chemistry</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>456</th>\n",
              "      <td>0812-1137</td>\n",
              "      <td>Biointerface Research in Applied Chemistry</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>533</th>\n",
              "      <td>0973-7294</td>\n",
              "      <td>International Journal of Tomography &amp; Simulation</td>\n",
              "      <td>CESER Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>534</th>\n",
              "      <td>0973-7308</td>\n",
              "      <td>International Journal of Ecology &amp; Development</td>\n",
              "      <td>CESER Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>535</th>\n",
              "      <td>0973-7537</td>\n",
              "      <td>International Journal of Ecological Economics ...</td>\n",
              "      <td>CESER Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>536</th>\n",
              "      <td>0973-7545</td>\n",
              "      <td>International Journal of Applied Mathematics a...</td>\n",
              "      <td>CESER Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>548</th>\n",
              "      <td>0974-0910</td>\n",
              "      <td>Journal of Cell and Tissue Research</td>\n",
              "      <td>CTR Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>562</th>\n",
              "      <td>0974-5718</td>\n",
              "      <td>International Journal of Mathematics &amp; Computa...</td>\n",
              "      <td>CESER Publications</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>578</th>\n",
              "      <td>0974-8998</td>\n",
              "      <td>Journal of Networking Technology</td>\n",
              "      <td>Digital Information Research Foundation (DIRF)</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>596</th>\n",
              "      <td>0975-2293</td>\n",
              "      <td>International Journal of Computer Networks &amp; C...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>597</th>\n",
              "      <td>0975-2307</td>\n",
              "      <td>International Journal of Network Security &amp; It...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>616</th>\n",
              "      <td>0975-427X</td>\n",
              "      <td>Asian Journal of Chemistry</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>618</th>\n",
              "      <td>0975-4660</td>\n",
              "      <td>International Journal of Computer Science &amp; In...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>619</th>\n",
              "      <td>0975-4679</td>\n",
              "      <td>International Journal of Wireless &amp; Mobile Net...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>627</th>\n",
              "      <td>0975-5926</td>\n",
              "      <td>International Journal of Managing Information ...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>0975-5934</td>\n",
              "      <td>The International Journal of Multimedia &amp; Its ...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>629</th>\n",
              "      <td>0975-5985</td>\n",
              "      <td>International Journal of Database Management S...</td>\n",
              "      <td>AIRCC Publishing Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13107</th>\n",
              "      <td>6095-2015</td>\n",
              "      <td>International Journal of Application or Innova...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13109</th>\n",
              "      <td>6633-2016</td>\n",
              "      <td>International Journal of Engineering Trends an...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13110</th>\n",
              "      <td>6755-4183</td>\n",
              "      <td>International Journal of Recent Scientific Res...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13111</th>\n",
              "      <td>6776-9071</td>\n",
              "      <td>International Journal of Computer Applications</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13112</th>\n",
              "      <td>7105-2016</td>\n",
              "      <td>International Journal of Engineering Research ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13113</th>\n",
              "      <td>7195-9290</td>\n",
              "      <td>Advances in Biomedicine and Pharmacy</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13114</th>\n",
              "      <td>7256-3713</td>\n",
              "      <td>International Journal of Research GRANTHAALAYAH</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13115</th>\n",
              "      <td>7258-7266</td>\n",
              "      <td>Genetics and Molecular Research</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13116</th>\n",
              "      <td>7356-2016</td>\n",
              "      <td>International Journal of Recent Scientific Res...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13117</th>\n",
              "      <td>7494-2016</td>\n",
              "      <td>International Journal of Engineering and Appli...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13118</th>\n",
              "      <td>7582-2014</td>\n",
              "      <td>International Journal for Research in Emerging...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13119</th>\n",
              "      <td>7801-2016</td>\n",
              "      <td>International Journal of Engineering Sciences ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13123</th>\n",
              "      <td>7901-2015</td>\n",
              "      <td>International Journal of Development Research</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13124</th>\n",
              "      <td>7937-2015</td>\n",
              "      <td>International Journal of Advance Research and ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13126</th>\n",
              "      <td>8140-2016</td>\n",
              "      <td>International Journal of Innovative and Applie...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13128</th>\n",
              "      <td>8244-8750</td>\n",
              "      <td>International Journal of Computer Engineering ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13129</th>\n",
              "      <td>8247-2016</td>\n",
              "      <td>International Journal of Emerging Research in ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13131</th>\n",
              "      <td>8455-2015</td>\n",
              "      <td>International Journal of Indian Psychology</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13132</th>\n",
              "      <td>8487-6366</td>\n",
              "      <td>Canadian International Journal of Science and ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13133</th>\n",
              "      <td>8555-2016</td>\n",
              "      <td>American Journal of Innovative Research and Ap...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13136</th>\n",
              "      <td>9001-2008</td>\n",
              "      <td>International Journal of Engineering Technolog...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13138</th>\n",
              "      <td>9134-2016</td>\n",
              "      <td>International Journal of Computer Engineering ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13139</th>\n",
              "      <td>9138-2015</td>\n",
              "      <td>International Journal of Engineering Sciences ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13140</th>\n",
              "      <td>9341-2016</td>\n",
              "      <td>International Journal of New Technology and Re...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13141</th>\n",
              "      <td>9377-2016</td>\n",
              "      <td>International Journal of Current Advanced Rese...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13143</th>\n",
              "      <td>9521-5978</td>\n",
              "      <td>International Journal of Dental and Health Sci...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13144</th>\n",
              "      <td>9680-5667</td>\n",
              "      <td>International Journal of Innovative Computer S...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13145</th>\n",
              "      <td>9681-2016</td>\n",
              "      <td>International Journal for Research in Applied ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13147</th>\n",
              "      <td>9826-4263</td>\n",
              "      <td>Advances in Biomedicine and Pharmacy</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13148</th>\n",
              "      <td>9933-2016</td>\n",
              "      <td>International Journal for Development of Compu...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1133 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            issn  ...                              url\n",
              "132    0053-0016  ...                              NaN\n",
              "133    0053-0017  ...                              NaN\n",
              "134    0053-0018  ...                              NaN\n",
              "135    0053-0019  ...                              NaN\n",
              "136    0053-0020  ...                              NaN\n",
              "137    0053-0021  ...                              NaN\n",
              "138    0053-0022  ...                              NaN\n",
              "139    0053-0023  ...                              NaN\n",
              "140    0053-0024  ...                              NaN\n",
              "141    0053-0025  ...                              NaN\n",
              "142    0053-0026  ...                              NaN\n",
              "435    0719-4986  ...  http://www.biodiversnathist.com\n",
              "454    0812-1126  ...                              NaN\n",
              "455    0812-1135  ...                              NaN\n",
              "456    0812-1137  ...                              NaN\n",
              "533    0973-7294  ...                              NaN\n",
              "534    0973-7308  ...                              NaN\n",
              "535    0973-7537  ...                              NaN\n",
              "536    0973-7545  ...                              NaN\n",
              "548    0974-0910  ...                              NaN\n",
              "562    0974-5718  ...                              NaN\n",
              "578    0974-8998  ...                              NaN\n",
              "596    0975-2293  ...                              NaN\n",
              "597    0975-2307  ...                              NaN\n",
              "616    0975-427X  ...                              NaN\n",
              "618    0975-4660  ...                              NaN\n",
              "619    0975-4679  ...                              NaN\n",
              "627    0975-5926  ...                              NaN\n",
              "628    0975-5934  ...                              NaN\n",
              "629    0975-5985  ...                              NaN\n",
              "...          ...  ...                              ...\n",
              "13107  6095-2015  ...                              NaN\n",
              "13109  6633-2016  ...                              NaN\n",
              "13110  6755-4183  ...                              NaN\n",
              "13111  6776-9071  ...                              NaN\n",
              "13112  7105-2016  ...                              NaN\n",
              "13113  7195-9290  ...                              NaN\n",
              "13114  7256-3713  ...                              NaN\n",
              "13115  7258-7266  ...                              NaN\n",
              "13116  7356-2016  ...                              NaN\n",
              "13117  7494-2016  ...                              NaN\n",
              "13118  7582-2014  ...                              NaN\n",
              "13119  7801-2016  ...                              NaN\n",
              "13123  7901-2015  ...                              NaN\n",
              "13124  7937-2015  ...                              NaN\n",
              "13126  8140-2016  ...                              NaN\n",
              "13128  8244-8750  ...                              NaN\n",
              "13129  8247-2016  ...                              NaN\n",
              "13131  8455-2015  ...                              NaN\n",
              "13132  8487-6366  ...                              NaN\n",
              "13133  8555-2016  ...                              NaN\n",
              "13136  9001-2008  ...                              NaN\n",
              "13138  9134-2016  ...                              NaN\n",
              "13139  9138-2015  ...                              NaN\n",
              "13140  9341-2016  ...                              NaN\n",
              "13141  9377-2016  ...                              NaN\n",
              "13143  9521-5978  ...                              NaN\n",
              "13144  9680-5667  ...                              NaN\n",
              "13145  9681-2016  ...                              NaN\n",
              "13147  9826-4263  ...                              NaN\n",
              "13148  9933-2016  ...                              NaN\n",
              "\n",
              "[1133 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "chva3DbpMZIt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#We don't want to do anything about the nulls just yet. However, they are being counted towards our duplicate values.\n",
        "#To work around this, we can do a groupby and size on our duplicated columns and add an occurrence count column.\n",
        "\n",
        "\n",
        "#Let's make it a new dataframe for better access\n",
        "jour_pub_url_group = df.groupby(['journal_name','pub_name','url']).size().reset_index().\\\n",
        "                    rename(columns={0:'occurrence_count'})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yYSQ4wh6M3_F",
        "colab_type": "code",
        "outputId": "8fa0c0d4-f836-4817-bc3b-876fb02d815f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        }
      },
      "source": [
        "# Lets see how many values are duplicated across journal_name,pub_name and url\n",
        "\n",
        "\n",
        "jour_pub_url_group[jour_pub_url_group['occurrence_count']>1]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>journal_name</th>\n",
              "      <th>pub_name</th>\n",
              "      <th>url</th>\n",
              "      <th>occurrence_count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>749</th>\n",
              "      <td>Biodiversity and Natural History</td>\n",
              "      <td>Centro de Estudios en Biodiversidad</td>\n",
              "      <td>http://www.biodiversnathist.com</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         journal_name  ... occurrence_count\n",
              "749  Biodiversity and Natural History  ...                2\n",
              "\n",
              "[1 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BHA4fo2sNNYs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now let's see how many values are duplicated in the entire dataset\n",
        "\n",
        "dupe_check_all_df= df.groupby(df.columns.tolist()).size().reset_index().\\\n",
        "                       rename(columns={0:'occurrence_count'})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xj7VcchXlC-8",
        "colab_type": "code",
        "outputId": "3c969b6a-abb8-4ee0-9712-2805573c5d20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# To get the total number of dupes, we can subtract the shape of our data from the occurence count total\n",
        "\n",
        "dupe_check_all_df['occurrence_count'].sum() - dupe_check_all_df.shape[0]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ztp-a7Hy_sgT",
        "colab_type": "text"
      },
      "source": [
        "There is a duplicate value in the journal_name, pub_name, and url columns. However, I am in no way  a subject matter expert in academic journals, so  I don't want to assume that this means that the data is inherently wrong. \n",
        "Since the issn column can be considered a unique identifier and does not contain duplicates, I believe the right move is to continue working with the data as is and consult the owners of the dataset later for clarification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0kYorVOtmxNp",
        "colab_type": "text"
      },
      "source": [
        "## 1.c:  Standardization and String Matching"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RCgjQJtvwNbQ",
        "colab_type": "code",
        "outputId": "041c2bec-1be4-482d-c1b7-82a3243af586",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        }
      },
      "source": [
        "# From what we've already seen of the data, we can recognize that character case is inconsistent in pub_name and category columns\n",
        "# This can be quickly fixed by converting all columns with object datatypes to lowercase\n",
        "\n",
        "df = df.apply(lambda x: x.str.lower() if(x.dtype == 'object') else x)\n",
        "\n",
        "df.head(2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>issn</th>\n",
              "      <th>journal_name</th>\n",
              "      <th>pub_name</th>\n",
              "      <th>is_hybrid</th>\n",
              "      <th>category</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0001-527x</td>\n",
              "      <td>acta biochimica polonica</td>\n",
              "      <td>acta biochimica polonica</td>\n",
              "      <td>0</td>\n",
              "      <td>molecular and cell biology</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0002-0397</td>\n",
              "      <td>africa spectrum</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        issn              journal_name  ...                    category  url\n",
              "0  0001-527x  acta biochimica polonica  ...  molecular and cell biology  NaN\n",
              "1  0002-0397           africa spectrum  ...                         NaN  NaN\n",
              "\n",
              "[2 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3pnzIKiPYa0K",
        "colab_type": "code",
        "outputId": "0067e897-694a-42fc-d61d-994a7af52da3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "#We can alse apply the same string functions to individual columns...\n",
        "\n",
        "\n",
        "df['journal_name'] = df['journal_name'].str.title() # Let's make journal_name title case\n",
        "                                                    # Careful with this one,it doesn't discriminate and will capitalize articles and prepositions\n",
        "\n",
        "df['issn']= df['issn'].str.upper() #And lets make issn uppercase \n",
        "\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>issn</th>\n",
              "      <th>journal_name</th>\n",
              "      <th>pub_name</th>\n",
              "      <th>is_hybrid</th>\n",
              "      <th>category</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0001-527X</td>\n",
              "      <td>Acta Biochimica Polonica</td>\n",
              "      <td>acta biochimica polonica</td>\n",
              "      <td>0</td>\n",
              "      <td>molecular and cell biology</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0002-0397</td>\n",
              "      <td>Africa Spectrum</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0003-0090</td>\n",
              "      <td>Bulletin Of The American Museum Of Natural His...</td>\n",
              "      <td>amer museum natural history</td>\n",
              "      <td>0</td>\n",
              "      <td>ecology and evolution</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0003-5521</td>\n",
              "      <td>L'Anthropologie</td>\n",
              "      <td>elsevier</td>\n",
              "      <td>1</td>\n",
              "      <td>anthropology</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0004-1254</td>\n",
              "      <td>Arhiv Za Higijenu Rada I Toksikologiju-Archive...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        issn  ...  url\n",
              "0  0001-527X  ...  NaN\n",
              "1  0002-0397  ...  NaN\n",
              "2  0003-0090  ...  NaN\n",
              "3  0003-5521  ...  NaN\n",
              "4  0004-1254  ...  NaN\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Z54Xuw8wNxI",
        "colab_type": "code",
        "outputId": "117037e5-763d-480d-cb9f-4b120baf0a96",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 998
        }
      },
      "source": [
        " df['category'].unique()\n",
        "\n",
        "# Looking at the unique values of the category column we can see that we still have some problems with inconsistent delimiter use\n",
        "# We want to standardize the data and make it more consistent"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['molecular and cell biology', nan, 'ecology and evolution',\n",
              "       'anthropology', 'robotics', 'physics and chemistry', 'psychology',\n",
              "       'medicine', 'neuroscience', 'veterinary', 'economics', 'law',\n",
              "       'gastroenterology', 'plant biology', 'orthopedics', 'agriculture',\n",
              "       'linguistics', 'history and philosophy of science', 'mathematics',\n",
              "       'oncology', 'sports medicine', 'food science', 'psychiatry',\n",
              "       'science', 'radiology', 'urology', 'environmental health',\n",
              "       'social sciences', 'ophthalmology', 'marketing',\n",
              "       'astronomy and astrophysics', 'operations research',\n",
              "       'infectious diseases', 'literary studies', 'plastic surgery',\n",
              "       'energy', 'structural engineering',\n",
              "       'philosophy. psychology. religion', 'fine arts',\n",
              "       'music and books on music', 'history america',\n",
              "       'geography. anthropology. recreation', 'political science',\n",
              "       'language and literature', 'technology', 'education',\n",
              "       'general works', 'history (general) and history of europe',\n",
              "       'technology | law', 'history america | social sciences',\n",
              "       'auxiliary sciences of history',\n",
              "       'political science | social sciences', 'computer science',\n",
              "       'information & library sciences',\n",
              "       'library and information science', 'education | technology',\n",
              "       'medicine | science', 'circuits', 'information science',\n",
              "       'geotechnology', 'sociology', 'nephrology', 'dentistry',\n",
              "       'rheumatology', 'pharmacology', 'agriculture | science',\n",
              "       'medicine | social sciences',\n",
              "       'fine arts | geography. anthropology. recreation', 'informatics',\n",
              "       'communication & media studies', 'education | science',\n",
              "       'military science', 'technology | social sciences',\n",
              "       'law | social sciences', 'naval science', 'technology | science',\n",
              "       'agriculture | technology', 'law | political science',\n",
              "       'languages & linguistics',\n",
              "       'geography. anthropology. recreation | science',\n",
              "       'philosophy. psychology. religion | social sciences',\n",
              "       'wood products', 'fine arts | language and literature',\n",
              "       'technology | language and literature', 'fine arts | science',\n",
              "       'literature', 'policy and law',\n",
              "       'education | political science | social sciences',\n",
              "       'agriculture | medicine', 'business & management',\n",
              "       'medicine | philosophy. psychology. religion',\n",
              "       'education | social sciences',\n",
              "       'education | language and literature', 'law | medicine',\n",
              "       'general science & technology', 'nursing',\n",
              "       'language and literature | social sciences',\n",
              "       'agriculture | geography. anthropology. recreation',\n",
              "       'scholarly publishing', 'agriculture | technology | science',\n",
              "       'technology | medicine', 'technology | medicine | science',\n",
              "       'philosophy. psychology. religion | science',\n",
              "       'education | philosophy. psychology. religion',\n",
              "       'technology | philosophy. psychology. religion',\n",
              "       'education | medicine', 'scientific equipment', 'dynamics',\n",
              "       'education | geography. anthropology. recreation',\n",
              "       'arts & humanities', 'architecture',\n",
              "       'military science | political science', 'arts and humanities'],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T_lxpP15Rz7j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Lets replace 'and' with '&'\n",
        "df['category'].replace(to_replace='and', value='&', regex=True,inplace=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iX6qQ-zRwOFS",
        "colab_type": "code",
        "outputId": "48a4b2b9-857f-4ddc-8777-1b7f303ebb28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "source": [
        "# Now back to those pesky delimiters ...\n",
        "\n",
        "# Overall, there are 12 metacharacters you should look out for  . ^ $ * + ? { } [ ] \\ | ( )\n",
        "#Both the pipe(|) and period(.) are considered metacharacters because they have special meaning.\n",
        " \n",
        "# Regular expressions use the backslash character ('\\') to indicate special forms or to allow special characters to be used without invoking their meaning. \n",
        "\n",
        "# This conflicts with Python’s usage of the same character for the same purpose in string literals and can become a pain when backslash\n",
        "# proliferation makes things unreadable\n",
        "\n",
        "print('ESCAPING WITH BACKSLASHES\\n')\n",
        "print('ROW CONTAINS PIPE \\n', df['category'].str.contains('\\\\|').sum())\n",
        "print('ROW CONTAINS PERIOD \\n', df['category'].str.contains('\\\\.').sum())\n",
        "print('ROW CONTAINS BACKSLASH \\n', df['category'].str.contains('\\\\\\\\').sum()) #This is when the backslashes really scale up\n",
        "print('ROW CONTAINS OPEN PARENTHESIS \\n', df['category'].str.contains('\\\\(').sum())\n",
        "print('ROW CONTAINS CLOSE PARENTHESIS \\n', df['category'].str.contains('\\\\)').sum())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ESCAPING WITH BACKSLASHES\n",
            "\n",
            "ROW CONTAINS PIPE \n",
            " 102\n",
            "ROW CONTAINS PERIOD \n",
            " 595\n",
            "ROW CONTAINS BACKSLASH \n",
            " 0\n",
            "ROW CONTAINS OPEN PARENTHESIS \n",
            " 128\n",
            "ROW CONTAINS CLOSE PARENTHESIS \n",
            " 128\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4t4KZJEQCGL0",
        "colab_type": "code",
        "outputId": "b27cdd25-cb23-4a0b-fa54-194c634d119d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "source": [
        "# A cleaner solution for this is to use raw string notation in addition to a backslash by adding an r outside of the quoted expression.\n",
        "print('ESCAPING WITH RAW STRING NOTATION \\n')\n",
        "print('ROW CONTAINS PIPE \\n', df['category'].str.contains(r'\\|').sum())\n",
        "print('ROW CONTAINS PERIOD \\n', df['category'].str.contains(r'\\.').sum())\n",
        "print('ROW CONTAINS BACKSLASH \\n', df['category'].str.contains(r'\\\\').sum())\n",
        "print('ROW CONTAINS OPEN PARENTHESIS \\n', df['category'].str.contains(r'\\(').sum())\n",
        "print('ROW CONTAINS CLOSE PARENTHESIS \\n', df['category'].str.contains(r'\\)').sum())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ESCAPING WITH RAW STRING NOTATION \n",
            "\n",
            "ROW CONTAINS PIPE \n",
            " 102\n",
            "ROW CONTAINS PERIOD \n",
            " 595\n",
            "ROW CONTAINS BACKSLASH \n",
            " 0\n",
            "ROW CONTAINS OPEN PARENTHESIS \n",
            " 128\n",
            "ROW CONTAINS CLOSE PARENTHESIS \n",
            " 128\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jBSYwIb87nTZ",
        "colab_type": "code",
        "outputId": "96d02aa2-50a1-48d9-a376-8acda829855f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#Intuitively, we can see that parenthesis are being used differently than pipes in the context of this dataset. \n",
        "#Because of this, we will leave them out of the special chars replacement.\n",
        "\n",
        "\n",
        "special_chars=r'\\.|\\^|\\$|\\*|\\+|\\?|\\{|\\}|\\[|\\]|\\\\|\\|' # Note that with the backslash, we're searching for the literal | character \n",
        "                                                     # Without the backslash we're using | as an OR statement\n",
        "\n",
        "  \n",
        "len(df[df['category'].str.contains(special_chars,na=False,regex=True)])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "684"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "udfBOUQkVNn3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Lets replace the special characters with an underscore.\n",
        "\n",
        "df['category'].replace(to_replace=special_chars, value='_', regex=True, inplace=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h98MKs4uiq9p",
        "colab_type": "code",
        "outputId": "31a3d070-7fa4-465f-d698-ec1721240c1f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "df.category"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        molecular & cell biology\n",
              "1                             NaN\n",
              "2             ecology & evolution\n",
              "3                    anthropology\n",
              "4                             NaN\n",
              "5                             NaN\n",
              "6                        robotics\n",
              "7                             NaN\n",
              "8                             NaN\n",
              "9             physics & chemistry\n",
              "10                     psychology\n",
              "11                            NaN\n",
              "12                       medicine\n",
              "13                   neuroscience\n",
              "14                            NaN\n",
              "15                     veterinary\n",
              "16            physics & chemistry\n",
              "17                       medicine\n",
              "18                            NaN\n",
              "19                       medicine\n",
              "20                            NaN\n",
              "21                      economics\n",
              "22                            NaN\n",
              "23                            NaN\n",
              "24                            law\n",
              "25                            NaN\n",
              "26                            NaN\n",
              "27               gastroenterology\n",
              "28                            NaN\n",
              "29                  plant biology\n",
              "                   ...           \n",
              "13119                         NaN\n",
              "13120                         NaN\n",
              "13121                         NaN\n",
              "13122                         NaN\n",
              "13123                         NaN\n",
              "13124                         NaN\n",
              "13125                         NaN\n",
              "13126                         NaN\n",
              "13127                         NaN\n",
              "13128                         NaN\n",
              "13129                         NaN\n",
              "13130                         NaN\n",
              "13131                         NaN\n",
              "13132                         NaN\n",
              "13133                         NaN\n",
              "13134                         NaN\n",
              "13135                    medicine\n",
              "13136                         NaN\n",
              "13137                         NaN\n",
              "13138                         NaN\n",
              "13139                         NaN\n",
              "13140                         NaN\n",
              "13141                         NaN\n",
              "13142                         NaN\n",
              "13143                         NaN\n",
              "13144                         NaN\n",
              "13145                         NaN\n",
              "13146                         NaN\n",
              "13147                         NaN\n",
              "13148                         NaN\n",
              "Name: category, Length: 13149, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "96xJ79DewQci",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now lets clean up the whitespace before and after the underscore\n",
        "\n",
        "df['category']=df['category'].replace('_ ', '_',regex=True).replace(' _','_',regex=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uoqJ6KxUwQu_",
        "colab_type": "code",
        "outputId": "17921231-ea0b-4885-bf93-27a9ec1b5843",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 925
        }
      },
      "source": [
        "df['category'].unique()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['molecular & cell biology', nan, 'ecology & evolution',\n",
              "       'anthropology', 'robotics', 'physics & chemistry', 'psychology',\n",
              "       'medicine', 'neuroscience', 'veterinary', 'economics', 'law',\n",
              "       'gastroenterology', 'plant biology', 'orthopedics', 'agriculture',\n",
              "       'linguistics', 'history & philosophy of science', 'mathematics',\n",
              "       'oncology', 'sports medicine', 'food science', 'psychiatry',\n",
              "       'science', 'radiology', 'urology', 'environmental health',\n",
              "       'social sciences', 'ophthalmology', 'marketing',\n",
              "       'astronomy & astrophysics', 'operations research',\n",
              "       'infectious diseases', 'literary studies', 'plastic surgery',\n",
              "       'energy', 'structural engineering',\n",
              "       'philosophy_psychology_religion', 'fine arts',\n",
              "       'music & books on music', 'history america',\n",
              "       'geography_anthropology_recreation', 'political science',\n",
              "       'language & literature', 'technology', 'education',\n",
              "       'general works', 'history (general) & history of europe',\n",
              "       'technology_law', 'history america_social sciences',\n",
              "       'auxiliary sciences of history',\n",
              "       'political science_social sciences', 'computer science',\n",
              "       'information & library sciences', 'library & information science',\n",
              "       'education_technology', 'medicine_science', 'circuits',\n",
              "       'information science', 'geotechnology', 'sociology', 'nephrology',\n",
              "       'dentistry', 'rheumatology', 'pharmacology', 'agriculture_science',\n",
              "       'medicine_social sciences',\n",
              "       'fine arts_geography_anthropology_recreation', 'informatics',\n",
              "       'communication & media studies', 'education_science',\n",
              "       'military science', 'technology_social sciences',\n",
              "       'law_social sciences', 'naval science', 'technology_science',\n",
              "       'agriculture_technology', 'law_political science',\n",
              "       'languages & linguistics',\n",
              "       'geography_anthropology_recreation_science',\n",
              "       'philosophy_psychology_religion_social sciences', 'wood products',\n",
              "       'fine arts_language & literature',\n",
              "       'technology_language & literature', 'fine arts_science',\n",
              "       'literature', 'policy & law',\n",
              "       'education_political science_social sciences',\n",
              "       'agriculture_medicine', 'business & management',\n",
              "       'medicine_philosophy_psychology_religion',\n",
              "       'education_social sciences', 'education_language & literature',\n",
              "       'law_medicine', 'general science & technology', 'nursing',\n",
              "       'language & literature_social sciences',\n",
              "       'agriculture_geography_anthropology_recreation',\n",
              "       'scholarly publishing', 'agriculture_technology_science',\n",
              "       'technology_medicine', 'technology_medicine_science',\n",
              "       'philosophy_psychology_religion_science',\n",
              "       'education_philosophy_psychology_religion',\n",
              "       'technology_philosophy_psychology_religion', 'education_medicine',\n",
              "       'scientific equipment', 'dynamics',\n",
              "       'education_geography_anthropology_recreation', 'arts & humanities',\n",
              "       'architecture', 'military science_political science'], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pUGid4gFPe5a",
        "colab_type": "text"
      },
      "source": [
        "# Part 2: Handling Missing Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zmLWkdjnK_-T",
        "colab_type": "text"
      },
      "source": [
        "## 2.1 On Missingness \n",
        "\n",
        "Resource: http://www.stat.columbia.edu/~gelman/arm/missing.pdf\n",
        "\n",
        "\n",
        "1. **Missingness completely at random (MCAR)**. A variable is missing completely at random if the probability of missingness is the same for all units, for example, if each survey respondent decides whether to answer the “earnings” question by rolling a die and refusing to answer if a “6” shows up. If data are missing completely at\n",
        "random, then throwing out cases with missing data does not bias your inferences.\n",
        "\n",
        "2. **Missingness at random (MAR)**. Missing at random means that the propensity for a data point to be missing is not related to the missing data, but it is related to some of the observed data.\n",
        "Most missingness is not completely at random, as can be seen from the data themselves. A more general assumption, missing at random, is that the probability a variable\n",
        "is missing depends only on available information. Thus, if sex, race, education, and age are recorded for all the people in the survey, then “earnings” is missing at random if the probability of nonresponse to this question depends only on these other, fully recorded variables. It is often reasonable to model this process as a logistic regression, where the outcome variable equals 1 for observed cases\n",
        "and 0 for missing. When an outcome variable is missing at random, it is acceptable to exclude the missing cases (that is, to treat them as NA’s), as long as the regression controls for all the variables that affect the probability of missingness. Thus, any model\n",
        "for earnings would have to include predictors for ethnicity, to avoid nonresponse bias. This missing-at-random assumption (a more formal version of which is sometimes called the ignorability assumption) in the missing-data framework is the basically same sort of assumption as ignorability in the causal framework. Both require that sufficient information has been collected that we can “ignore” the\n",
        "assignment mechanism (assignment to treatment, assignment to nonresponse).\n",
        "3. **Missingness not at random (MNAR)**: \n",
        ". Missingness is no longer “at random” if it depends on information that has not been recorded and this information also predicts the missing values. For example, suppose that “surly”\n",
        "people are less likely to respond to the earnings question, surliness is predictive\n",
        "of earnings, and “surliness” is unobserved. Or, suppose that people with college\n",
        "degrees are less likely to reveal their earnings, having a college degree is predictive of earnings, and there is also some nonresponse to the education question.\n",
        "Then, once again, earnings are not missing at random. If missingness is not at random, it must be explicitly modeled, or else you must\n",
        "accept some bias in your inferences. There are two types:\n",
        "\n",
        "*   **Missingness that depends on unobserved predictors**. A familiar example from medical studies is that if a particular treatment causes\n",
        "discomfort, a patient is more likely to drop out of the study. This missingness is\n",
        "not at random (unless “discomfort” is measured and observed for all patients).\n",
        "*   **Missingness that depends on the missing value itself**. Finally, a particularly difficult situation arises when the probability of missingness depends on the (potentially missing) variable itself. For example, suppose that people with higher earnings are less likely to reveal them. In the extreme case (for example, all persons earning more than $100,000 refuse to respond), this is called censoring, but\n",
        "even the probabilistic case causes difficulty.\n",
        "\n",
        "In the first two cases, it is safe to remove the data with missing values depending upon their occurrences. \n",
        "\n",
        "In the third case (MNAR), removing observations with missing values can produce a bias in the model. \n",
        "\n",
        "So we have to be really careful before removing observations.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CvskEJDouBea",
        "colab_type": "text"
      },
      "source": [
        "##2.2 Choices, Choices\n",
        "\n",
        "There are 3 choices for missing data:\n",
        "\n",
        "**1.   Imputation: Fill in the missing data**\n",
        "*   User provided data\n",
        "*   Summary statistic (Be careful here: mean, median, mode, etc. Use median if you have outliers)\n",
        "*   Modeled or imputed data\n",
        "*   Data present in dataset\n",
        "\n",
        "**2. Deletion: Drop rows or columns with missing data**\n",
        "*   Drop all \n",
        "*   Drop some, based on conditions or a threshold\n",
        "\n",
        "**3. Leave it as is**\n",
        "\n",
        "**When should one fill in data, drop data, or leave it alone? **\n",
        "\n",
        "It depends. . . .\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yuMY7GAAVBEO",
        "colab_type": "text"
      },
      "source": [
        "![Handling Missing Data](https://cdn-images-1.medium.com/max/1600/1*_RA3mCS30Pr0vUxbp25Yxw.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vFPIcydCBWcW",
        "colab_type": "text"
      },
      "source": [
        "##  2.3 Let's Get to Work\n",
        "\n",
        "### 2.3.1 Data Quality Check\n",
        "Let's check out our data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gcgrRVtVBs65",
        "colab_type": "code",
        "outputId": "c1264fdd-6257-4531-efeb-427c5ff77b88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "#Check out the dataset\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>issn</th>\n",
              "      <th>journal_name</th>\n",
              "      <th>pub_name</th>\n",
              "      <th>is_hybrid</th>\n",
              "      <th>category</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0001-527X</td>\n",
              "      <td>Acta Biochimica Polonica</td>\n",
              "      <td>acta biochimica polonica</td>\n",
              "      <td>0</td>\n",
              "      <td>molecular &amp; cell biology</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0002-0397</td>\n",
              "      <td>Africa Spectrum</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0003-0090</td>\n",
              "      <td>Bulletin Of The American Museum Of Natural His...</td>\n",
              "      <td>amer museum natural history</td>\n",
              "      <td>0</td>\n",
              "      <td>ecology &amp; evolution</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0003-5521</td>\n",
              "      <td>L'Anthropologie</td>\n",
              "      <td>elsevier</td>\n",
              "      <td>1</td>\n",
              "      <td>anthropology</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0004-1254</td>\n",
              "      <td>Arhiv Za Higijenu Rada I Toksikologiju-Archive...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        issn  ...  url\n",
              "0  0001-527X  ...  NaN\n",
              "1  0002-0397  ...  NaN\n",
              "2  0003-0090  ...  NaN\n",
              "3  0003-5521  ...  NaN\n",
              "4  0004-1254  ...  NaN\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1J50Q6Z3gu0x",
        "colab_type": "text"
      },
      "source": [
        "## 2.4 Assessing Missingness\n",
        "\n",
        "How much missingness is there?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p4Y0QPxzBt9L",
        "colab_type": "code",
        "outputId": "e0256f81-0af4-4a7e-e3ed-cb0d04066695",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 144
        }
      },
      "source": [
        "#Check out the column types\n",
        "df.dtypes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "issn            object\n",
              "journal_name    object\n",
              "pub_name        object\n",
              "is_hybrid        int64\n",
              "category        object\n",
              "url             object\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "9f76dd2d-6ebc-4791-d564-47fc4085e9ed",
        "id": "kepOfpZRifzF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Let's find out which columns have missing data\n",
        "missing_columns = list(df.columns[df.isnull().any()])\n",
        "missing_columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['pub_name', 'category', 'url']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "9203627c-628d-4b48-e897-a2fc2a6f59cd",
        "id": "AYcv9T2-ifzK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "#Let's find out how many missing values are in each column\n",
        "\n",
        "for col in missing_columns:\n",
        "    num_missing = df[df[col].isnull() == True].shape[0]\n",
        "    print('number missing for column {}: {}'.format(col, \n",
        "                                                    num_missing))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "number missing for column pub_name: 1470\n",
            "number missing for column category: 6331\n",
            "number missing for column url: 6788\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "cda6f0e6-6a69-4f51-ba9e-cafb884978ec",
        "id": "MQuT5WHVifzN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "#Percentage of missingness per column\n",
        "for col in missing_columns:\n",
        "    percent_missing = df[df[col].isnull() == True].shape[0]  / df.shape[0] * 100\n",
        "    print('percent missing for column {}: {}%'.format(\n",
        "        col, percent_missing))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "percent missing for column pub_name: 11.179557380789413%\n",
            "percent missing for column category: 48.148148148148145%\n",
            "percent missing for column url: 51.62369761959085%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4g7iEmEFCmmi",
        "colab_type": "text"
      },
      "source": [
        "## 2.5 DELETION\n",
        "If you have a \"small\" amount of data missing:\n",
        "\n",
        "\n",
        "1.   **Listwise deletion**: remove ALL data for an observation that has at least 1 missing value. \n",
        "> *Problem: Meeting MCAR assumptions are rare; as a result most of time you'll produce biased parameters and estimates*\n",
        "\n",
        "\n",
        "2.   **Pairwise deletion**: Keep all data that isn't missing. Even though it increases the power of your analysis, you'll end up with different numbers of observations contributing to different parts of your model. This makes interpretation very difficult.\n",
        "\n",
        "2.   **Deleting Columns**: Keep all data that isn't missing. Even though it increases the power of your analysis, you'll end up with different numbers of observations contributing to different parts of your model. This makes interpretation very difficult.\n",
        "\n",
        "\n",
        "\n",
        "### 2.5.1 Remove Null Values Upon Reading in Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S4uTgdh7CxDc",
        "colab_type": "code",
        "outputId": "7ded0aef-32ee-457a-e29d-f2b37f450d1a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "#Call na_values to remove n/a values\n",
        "# Making a list of missing value types\n",
        "# Read in data using Pandas and quickly ELIMINATE missing values based on list of missing value types\n",
        "missing_values = [\"n/a\", \"na\", \"--\"]\n",
        "df = pd.read_csv(url, na_values = missing_values)\n",
        "df.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>journal_name</th>\n",
              "      <th>issn</th>\n",
              "      <th>citation_count_sum</th>\n",
              "      <th>paper_count_sum</th>\n",
              "      <th>avg_cites_per_paper</th>\n",
              "      <th>proj_ai</th>\n",
              "      <th>proj_ai_year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3d research</td>\n",
              "      <td>2092-6731</td>\n",
              "      <td>151.0</td>\n",
              "      <td>106.0</td>\n",
              "      <td>1.424528</td>\n",
              "      <td>0.290</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>aaps pharmscitech</td>\n",
              "      <td>1530-9932</td>\n",
              "      <td>2208.0</td>\n",
              "      <td>801.0</td>\n",
              "      <td>2.756554</td>\n",
              "      <td>0.665</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>abstract and applied analysis</td>\n",
              "      <td>1687-0409</td>\n",
              "      <td>3005.0</td>\n",
              "      <td>2923.0</td>\n",
              "      <td>1.028053</td>\n",
              "      <td>0.192</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>academic psychiatry</td>\n",
              "      <td>1545-7230</td>\n",
              "      <td>537.0</td>\n",
              "      <td>490.0</td>\n",
              "      <td>1.095918</td>\n",
              "      <td>0.208</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>academic questions</td>\n",
              "      <td>1936-4709</td>\n",
              "      <td>40.0</td>\n",
              "      <td>67.0</td>\n",
              "      <td>0.597015</td>\n",
              "      <td>0.097</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5</td>\n",
              "      <td>accreditation and quality assurance</td>\n",
              "      <td>1432-0517</td>\n",
              "      <td>255.0</td>\n",
              "      <td>331.0</td>\n",
              "      <td>0.770393</td>\n",
              "      <td>0.134</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>6</td>\n",
              "      <td>acoustics australia</td>\n",
              "      <td>1839-2571</td>\n",
              "      <td>30.0</td>\n",
              "      <td>25.0</td>\n",
              "      <td>1.200000</td>\n",
              "      <td>0.234</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>7</td>\n",
              "      <td>acrocephalus</td>\n",
              "      <td>2199-6067</td>\n",
              "      <td>9.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.600000</td>\n",
              "      <td>0.098</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>8</td>\n",
              "      <td>acta adriatica</td>\n",
              "      <td>1846-0453</td>\n",
              "      <td>28.0</td>\n",
              "      <td>37.0</td>\n",
              "      <td>0.756757</td>\n",
              "      <td>0.131</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>9</td>\n",
              "      <td>acta agriculturae slovenica</td>\n",
              "      <td>1854-1941</td>\n",
              "      <td>71.0</td>\n",
              "      <td>97.0</td>\n",
              "      <td>0.731959</td>\n",
              "      <td>0.125</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0                         journal_name  ... proj_ai  proj_ai_year\n",
              "0           0                          3d research  ...   0.290          2015\n",
              "1           1                    aaps pharmscitech  ...   0.665          2015\n",
              "2           2        abstract and applied analysis  ...   0.192          2015\n",
              "3           3                  academic psychiatry  ...   0.208          2015\n",
              "4           4                   academic questions  ...   0.097          2015\n",
              "5           5  accreditation and quality assurance  ...   0.134          2015\n",
              "6           6                  acoustics australia  ...   0.234          2015\n",
              "7           7                         acrocephalus  ...   0.098          2015\n",
              "8           8                       acta adriatica  ...   0.131          2015\n",
              "9           9          acta agriculturae slovenica  ...   0.125          2015\n",
              "\n",
              "[10 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hj5XUYCDDpiE",
        "colab_type": "code",
        "outputId": "e937623d-26d0-4d8e-abd8-e93b10c1b6a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3615, 8)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLVZg4QkhXNJ",
        "colab_type": "text"
      },
      "source": [
        "## 2.6 What to do with NaNs?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFlVXC1Hkwg5",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "If you need to fill in errors or blanks, use the ```fillna()``` and ```dropna()``` methods. \n",
        "\n",
        "It seems quick, but all manipulations of the data should be documented so you can explain them to someone at a later time.\n",
        "\n",
        "You could fill the NaNs with strings, or if they are numbers you could use the mean or the median value. There is a lot of debate on what do with missing or malformed data, and the correct answer is … it depends.\n",
        "\n",
        "You’ll have to use your best judgement and input from the people you’re working with on why removing or filling the data is the best approach.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r6Bj9UnKi4mq",
        "colab_type": "code",
        "outputId": "d227c4f7-c262-4276-8b8a-cac547080509",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        }
      },
      "source": [
        "#Now we see that 6788 values are filled in with 0\n",
        "df.url.value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-69-2a144facf085>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue_counts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m   5065\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_info_axis\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_can_hold_identifiers_and_holds_name\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5066\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5067\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5068\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5069\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: 'DataFrame' object has no attribute 'url'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pt2hY6ZykOZK",
        "colab_type": "text"
      },
      "source": [
        "### 2.6.1 Filling in Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "275cbc0a-1b36-49c2-c3fa-c7cc65fee0b9",
        "id": "nRIPekpgifzg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1128
        }
      },
      "source": [
        "# First, find the value counts of the URL column\n",
        "df.url.value_counts()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "https://www.springer.com/us/open-access/springer-open-choice                     14\n",
              "https://www.hindawi.com/journals/acp/                                             3\n",
              "http://www.degruyter.com/view/j/plua                                              2\n",
              "http://www.ddtjournal.org/                                                        2\n",
              "http://www.biogeosciences.net                                                     2\n",
              "http://www.springer.com/us/open-access/springer-open-choice                       2\n",
              "https://www.hindawi.com/journals/jqre/                                            2\n",
              "http://epress.lib.uts.edu.au/journals/index.php/AJCEB                             2\n",
              "http://www.kmuj.kmu.edu.pk/index                                                  2\n",
              "http://newvoices.org.au/                                                          2\n",
              "http://www.collabra.org                                                           2\n",
              "http://www.egms.de/dynamic/en/journals/zma/index.htm                              2\n",
              "http://www.biodiversnathist.com                                                   2\n",
              "http://efsupit.ro/                                                                2\n",
              "https://www.hindawi.com/journals/jcnc/                                            2\n",
              "http://www.lawcrimehistory.org/journal.htm                                        2\n",
              "http://www.scoliosisjournal.com/                                                  2\n",
              "http://www.ocl-journal.org/                                                       2\n",
              "https://www.hindawi.com/journals/jchem/                                           2\n",
              "http://www2.um.edu.uy/ingenieria/revista/                                         2\n",
              "http://gymnica.upol.cz                                                            2\n",
              "http://journalmedica.com                                                          1\n",
              "http://www.jbmethods.org                                                          1\n",
              "http://www.degruyter.com/view/j/aoter                                             1\n",
              "http://www.socpvs.org/wbp/index.php/wbp                                           1\n",
              "http://www.degruyter.com/view/j/bimo                                              1\n",
              "http://digitalcommons.cedarville.edu/musicalofferings                             1\n",
              "http://dergipark.ulakbim.gov.tr/ktd/                                              1\n",
              "https://revistas.uam.es/riejs                                                     1\n",
              "https://www.metodista.br/revistas/revistas-unimep/index.php/comunicacao/index     1\n",
              "                                                                                 ..\n",
              "http://journal.uny.ac.id/index.php/hsjpi                                          1\n",
              "http://siba-ese.unisalento.it/index.php/psychofenia                               1\n",
              "http://www.revistas.unal.edu.co/index.php/actabiol                                1\n",
              "http://ateliers.revues.org/                                                       1\n",
              "https://sites.google.com/site/1rvcta                                              1\n",
              "http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)2052-4412                    1\n",
              "https://www.hindawi.com/journals/an/                                              1\n",
              "http://www.barnboken.net                                                          1\n",
              "http://commons.erau.edu/jaaer/                                                    1\n",
              "http://jurnal.ugm.ac.id/buletinpsikologi                                          1\n",
              "http://rps.mui.ac.ir/index.php/jrps                                               1\n",
              "http://kmhj.ukma.edu.ua/                                                          1\n",
              "http://www.biologie.uvt.ro/annals.html                                            1\n",
              "http://www.e-enm.org/                                                             1\n",
              "http://demesci.hipatiapress.com                                                   1\n",
              "http://www.gw-unterricht.at                                                       1\n",
              "http://ijqr.net/                                                                  1\n",
              "http://www.ajts.org/                                                              1\n",
              "http://scialert.net/jindex.php?issn=1811-9778                                     1\n",
              "http://history-pages.kpi.ua/                                                      1\n",
              "http://www.apospublications.com                                                   1\n",
              "https://www.hindawi.com/journals/ijg/                                             1\n",
              "http://www.geologia-croatica.hr/index.html                                        1\n",
              "http://epress.lib.uts.edu.au/ojs/index.php/portal                                 1\n",
              "http://revistas.uis.edu.co/index.php/revistaion/index                             1\n",
              "http://www.degruyter.com/view/j/hacq                                              1\n",
              "http://palabraclave.unisabana.edu.co/index.php/palabraclave/index                 1\n",
              "http://jurnal.balithutmakassar.org/index.php/wallacea/index                       1\n",
              "https://www.hindawi.com/journals/art/                                             1\n",
              "http://www.journals.elsevier.com/the-saudi-dental-journal/                        1\n",
              "Name: url, Length: 6327, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RmSTHoJGBrwl",
        "colab_type": "code",
        "outputId": "65f48c18-e604-4cfa-e6ed-1eade17a1e8a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        }
      },
      "source": [
        "# The top missing values in the URL column\n",
        "df.url.value_counts().head()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "https://www.springer.com/us/open-access/springer-open-choice    14\n",
              "https://www.hindawi.com/journals/acp/                            3\n",
              "http://www.degruyter.com/view/j/plua                             2\n",
              "http://www.ddtjournal.org/                                       2\n",
              "http://www.biogeosciences.net                                    2\n",
              "Name: url, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1U0WSfu6hswW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Fill in missing data using the .fillna method\n",
        "# Let's fill in missing values with 0\n",
        "df.url = df.url.fillna(0)\n",
        "\n",
        "# Fill NaN with ' '\n",
        "#df['url'] = df['url'].fillna(' ')\n",
        "\n",
        "# Fill NaN with 999\n",
        "#df['url'] = df['url'].fillna(999)\n",
        "\n",
        "# Fill NaN with the mean of the column -- for numeric columns only!!\n",
        "#df['url'] = df['url'].fillna(df['url'].mean())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WWnKRqIcmhlI",
        "colab_type": "text"
      },
      "source": [
        "You can also propagate non-null values forward or backwards by using ```method=’pad’``` as the method argument. \n",
        "\n",
        "\n",
        "It will fill the next value in the dataframe with the previous non-NaN value. \n",
        "\n",
        "When you want to fill one value, use ```limit=1```.\n",
        "\n",
        "If you don't set a limit, you'll fill in the entire dataframe \n",
        "\n",
        "Let's create a dataframe to demo forward fill and backfilll:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zbg82Df0mgH3",
        "colab_type": "code",
        "outputId": "aa59f08f-6b7c-494f-8dda-b9561efa8b0e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        }
      },
      "source": [
        "missing_df = pd.DataFrame(data={'col1':[np.nan, np.nan, 23,3,14,5,13, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan]})\n",
        "missing_df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    col1\n",
              "0    NaN\n",
              "1    NaN\n",
              "2   23.0\n",
              "3    3.0\n",
              "4   14.0\n",
              "5    5.0\n",
              "6   13.0\n",
              "7    NaN\n",
              "8    NaN\n",
              "9    NaN\n",
              "10   NaN\n",
              "11   NaN\n",
              "12   NaN\n",
              "13   NaN\n",
              "14   NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sq1U_5lnmgAM",
        "colab_type": "code",
        "outputId": "a1997057-91b8-4fe4-e209-7f96db9fcdd2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 189
        }
      },
      "source": [
        "missing_df.fillna(method='pad', limit=1)\n",
        "# See that index 7 is filled in with index 6's value  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-2-c9881ecf46de>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmissing_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfillna\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'pad'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlimit\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;31m# See that index 7 is filled in with index 6's value\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'missing_df' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QjVN9tl2mf5s",
        "colab_type": "code",
        "outputId": "2714f3c2-30af-4f4d-9fe2-03bca1526ec1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        }
      },
      "source": [
        "# If you don't set a limit, you'll fill in the rest of the entire dataframe \n",
        "missing_df.fillna(method='pad')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-3886d96cbbca>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmissing_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfillna\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'pad'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'missing_df' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKGsO31cqMjy",
        "colab_type": "text"
      },
      "source": [
        "#### 2.6.2 Backfilling\n",
        "\n",
        "We are not limited to forward filling. We can  also backfill with bfill."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEk15-cbmfuv",
        "colab_type": "code",
        "outputId": "fdfae421-be2a-4e69-fa82-2cf2c11d4370",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        }
      },
      "source": [
        "#The first two values are NaN's. Let's backfill them\n",
        "missing_df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    col1\n",
              "0    NaN\n",
              "1    NaN\n",
              "2   23.0\n",
              "3    3.0\n",
              "4   14.0\n",
              "5    5.0\n",
              "6   13.0\n",
              "7    NaN\n",
              "8    NaN\n",
              "9    NaN\n",
              "10   NaN\n",
              "11   NaN\n",
              "12   NaN\n",
              "13   NaN\n",
              "14   NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9L0MPnfdqW8E",
        "colab_type": "code",
        "outputId": "901df401-d0fd-4f3f-d7c3-2740179d2132",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        }
      },
      "source": [
        "# Fill the first two NaN values with the first available value\n",
        "missing_df.fillna(method='bfill')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    col1\n",
              "0   23.0\n",
              "1   23.0\n",
              "2   23.0\n",
              "3    3.0\n",
              "4   14.0\n",
              "5    5.0\n",
              "6   13.0\n",
              "7    NaN\n",
              "8    NaN\n",
              "9    NaN\n",
              "10   NaN\n",
              "11   NaN\n",
              "12   NaN\n",
              "13   NaN\n",
              "14   NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s900C-Dlq6u-",
        "colab_type": "text"
      },
      "source": [
        "##2.7 Dropping Missing Data\n",
        "\n",
        "Use the ```dropna``` method to drop rows or columns with NaNs. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DzLOJm9MqixV",
        "colab_type": "code",
        "outputId": "e2569b5d-be94-48a2-9a15-3b8eeb519549",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        }
      },
      "source": [
        "# Drop any rows which have any nans\n",
        "missing_df.dropna()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   col1\n",
              "2  23.0\n",
              "3   3.0\n",
              "4  14.0\n",
              "5   5.0\n",
              "6  13.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gXVmNxhkrSRx",
        "colab_type": "code",
        "outputId": "6c4baee3-eef5-4656-b1c2-b46ed5817289",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 494
        }
      },
      "source": [
        "# Drop columns that have any nans\n",
        "missing_df.dropna(axis=1)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZnNir_E5sYnO",
        "colab_type": "text"
      },
      "source": [
        "###2.7.1 NaNs thresholding\n",
        "The parameter thresh=N requires that a column has at least N non-NaNs to survive. Think of this as the lower limit for missing data you will find acceptable in your columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PSf9_1ZqrWPC",
        "colab_type": "code",
        "outputId": "21bbee6d-6ac4-4c9d-dbc7-3a031c6e1c4f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        }
      },
      "source": [
        "#Use a threshold to drop NaNs\n",
        "# Only drop columns which have at least 20% non-NaNs\n",
        "missing_df.dropna(thresh=int(missing_df.shape[0] * .20), axis=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    col1\n",
              "0    NaN\n",
              "1    NaN\n",
              "2   23.0\n",
              "3    3.0\n",
              "4   14.0\n",
              "5    5.0\n",
              "6   13.0\n",
              "7    NaN\n",
              "8    NaN\n",
              "9    NaN\n",
              "10   NaN\n",
              "11   NaN\n",
              "12   NaN\n",
              "13   NaN\n",
              "14   NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "65_52uKn-RB9",
        "colab_type": "text"
      },
      "source": [
        "## 2.8 IMPUTATION\n",
        "\n",
        "###2.8.1 SIMPLE IMPUTATION"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeJKNYMWDqou",
        "colab_type": "text"
      },
      "source": [
        "Replace missing values, encoded as ```np.nan```, using the mean value of the columns (axis 0) that contain the missing values.\n",
        "\n",
        "Missing values can be imputed with a provided constant value, or using the statistics (mean, median or most frequent) of each column in which the missing values are located.\n",
        "\n",
        "####2.8.1.1 For Numeric NaNs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xmiiesERrmnn",
        "colab_type": "code",
        "outputId": "36aa163f-2e8f-4250-b195-62d1fb5df9f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        }
      },
      "source": [
        "#NUMERIC NaNs\n",
        "#Impute with the mean\n",
        "#This will look for all columns where we have NaN value and replace the NaN value with specified test statistic.\n",
        "\n",
        "from sklearn.impute import SimpleImputer\n",
        "imputed_data = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
        "#Change strategy to 'median', and for the mode: 'most_frequent' \n",
        "\n",
        "imputed_data.fit(missing_df) #use .fit\n",
        "\n",
        "missing_df = pd.DataFrame(imputed_data.transform(missing_df)) #use.transform\n",
        "missing_df\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>14.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>11.6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       0\n",
              "0   11.6\n",
              "1   11.6\n",
              "2   23.0\n",
              "3    3.0\n",
              "4   14.0\n",
              "5    5.0\n",
              "6   13.0\n",
              "7   11.6\n",
              "8   11.6\n",
              "9   11.6\n",
              "10  11.6\n",
              "11  11.6\n",
              "12  11.6\n",
              "13  11.6\n",
              "14  11.6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DOxDQQgSGhPU",
        "colab_type": "code",
        "outputId": "813ac404-08c0-46dd-9fbb-bbaa57399397",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "#Let's look at the parameters of the SimpleImputer\n",
        "imputed_data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SimpleImputer(add_indicator=False, copy=True, fill_value=None,\n",
              "              missing_values=nan, strategy='mean', verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOLMuT3NE2ZR",
        "colab_type": "text"
      },
      "source": [
        "####2.8.1.2 For Categorical or *String* Data\n",
        "\n",
        "The SimpleImputer class also supports categorical data represented as string values or pandas categoricals when using the 'most_frequent' or 'constant' strategy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V6cAxFsSCQuc",
        "colab_type": "code",
        "outputId": "04057e3a-7d6b-4527-a61a-717249c5d840",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 172
        }
      },
      "source": [
        "#Let's create a dataframe with categorical or string data\n",
        "categorical_df = pd.DataFrame([[\"a\", \"x\"],\n",
        "                               [np.nan, \"y\"],\n",
        "                               [\"a\", np.nan],\n",
        "                               [\"b\", \"y\"]], dtype=\"category\")\n",
        "\n",
        "#Let's take a look at this new dataframe\n",
        "categorical_df\n",
        "\n",
        "#the most frequent value in each column is 'a' and the 'y'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>x</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>y</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>a</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>y</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1\n",
              "0    a    x\n",
              "1  NaN    y\n",
              "2    a  NaN\n",
              "3    b    y"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gu7kNqzvCtiv",
        "colab_type": "code",
        "outputId": "7fd24b9f-c155-49c4-c6ea-1d0d6b75c233",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "#Replacing the NaNs with the most frequent observation\n",
        "cat_imp = SimpleImputer(strategy=\"most_frequent\")\n",
        "print(cat_imp.fit_transform(categorical_df))      "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['a' 'x']\n",
            " ['a' 'y']\n",
            " ['a' 'y']\n",
            " ['b' 'y']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ImIbkF14DFTh",
        "colab_type": "code",
        "outputId": "94fd6b26-16d3-4ebe-cb9f-e8577d33c0ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "\n",
        "cat_imp"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SimpleImputer(add_indicator=False, copy=True, fill_value=None,\n",
              "              missing_values=nan, strategy='most_frequent', verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoDixaU6Hb7n",
        "colab_type": "text"
      },
      "source": [
        "## 2.9 Now Let's Get Fancy: FancyImpute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xvvpcaYOP2U",
        "colab_type": "text"
      },
      "source": [
        "####2.9.1 Imputation Algorithms\n",
        "\n",
        "**SimpleFill**: Replaces missing entries with the mean or median of each column.\n",
        "\n",
        "**KNN**: Nearest neighbor imputations which weights samples using the mean squared difference on features for which two rows both have observed data.\n",
        "\n",
        "**IterativeImputer**: A strategy for imputing missing values by modeling each feature with missing values as a function of other features in a round-robin fashion.\n",
        "\n",
        "**SoftImpute**: Matrix completion by iterative soft thresholding of SVD decompositions. Inspired by the softImpute package for R, which is based on Spectral Regularization Algorithms for Learning Large Incomplete Matrices by Mazumder et. al.\n",
        "\n",
        "\n",
        "**MatrixFactorization**: Direct factorization of the incomplete matrix into low-rank U and V, with an L1 sparsity penalty on the elements of U and an L2 penalty on the elements of V. Solved by gradient descent.\n",
        "\n",
        "**NuclearNormMinimization**: Simple implementation of Exact Matrix Completion via Convex Optimization by Emmanuel Candes and Benjamin Recht using cvxpy. Too slow for large matrices.\n",
        "\n",
        "**BiScaler**: Iterative estimation of row/column means and standard deviations to get doubly normalized matrix. Not guaranteed to converge but works well in practice. Taken from Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6sUHgK92Baho",
        "colab_type": "text"
      },
      "source": [
        "### Implementing Multiple Imputation\n",
        "We are using the metric -- Mean Square Error.\n",
        "\n",
        "Let's run all the imputation algorithms to see which one gives us the lowest error. The goal is to have an MSE of zero. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-B6X0nsKDGP3",
        "colab_type": "code",
        "outputId": "3e4a07eb-327f-4b72-a775-f56b5568f022",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3011
        }
      },
      "source": [
        "#Let's run all the imputation algorithms and see which ones give us the lowest error\n",
        "\n",
        "import numpy as np\n",
        "from fancyimpute import (\n",
        "    BiScaler,\n",
        "    KNN,\n",
        "    IterativeImputer,\n",
        "    NuclearNormMinimization,\n",
        "    SoftImpute,\n",
        "    SimpleFill)\n",
        "\n",
        "n = 200\n",
        "m = 20\n",
        "inner_rank = 4\n",
        "X = np.dot(np.random.randn(n, inner_rank), np.random.randn(inner_rank, m))\n",
        "print(\"Mean squared element: %0.4f\" % (X ** 2).mean())\n",
        "\n",
        "# X is a data matrix which we're going to randomly drop entries from\n",
        "missing_mask = np.random.rand(*X.shape) < 0.1\n",
        "X_incomplete = X.copy()\n",
        "# missing entries indicated with NaN\n",
        "X_incomplete[missing_mask] = np.nan\n",
        "\n",
        "meanFill = SimpleFill(\"mean\")\n",
        "X_filled_mean = meanFill.fit_transform(X_incomplete)\n",
        "\n",
        "# Model each feature with missing values as a function of other features, and\n",
        "# use that estimate for imputation.\n",
        "X_filled_ii = IterativeImputer().fit_transform(X_incomplete)\n",
        "\n",
        "# Use 3 nearest rows which have a feature to fill in each row's missing features\n",
        "knnImpute = KNN(k=3)\n",
        "X_filled_knn = knnImpute.fit_transform(X_incomplete)\n",
        "\n",
        "# matrix completion using convex optimization to find low-rank solution\n",
        "# that still matches observed values. Slow!\n",
        "X_filled_nnm = NuclearNormMinimization().fit_transform(X_incomplete)\n",
        "\n",
        "# Instead of solving the nuclear norm objective directly, instead\n",
        "# induce sparsity using singular value thresholding\n",
        "softImpute = SoftImpute()\n",
        "\n",
        "# simultaneously normalizes the rows and columns of your observed data,\n",
        "# sometimes useful for low-rank imputation methods\n",
        "biscaler = BiScaler()\n",
        "\n",
        "# rescale both rows and columns to have zero mean and unit variance\n",
        "X_incomplete_normalized = biscaler.fit_transform(X_incomplete)\n",
        "\n",
        "X_filled_softimpute_normalized = softImpute.fit_transform(X_incomplete_normalized)\n",
        "X_filled_softimpute = biscaler.inverse_transform(X_filled_softimpute_normalized)\n",
        "\n",
        "X_filled_softimpute_no_biscale = softImpute.fit_transform(X_incomplete)\n",
        "\n",
        "meanfill_mse = ((X_filled_mean[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"meanFill MSE: %f\" % meanfill_mse)\n",
        "\n",
        "# print mean squared error for the four imputation methods above\n",
        "# print mean squared error for the three imputation methods above\n",
        "ii_mse = ((X_filled_ii[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"Iterative Imputer norm minimization MSE: %f\" % ii_mse)\n",
        "\n",
        "nnm_mse = ((X_filled_nnm[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"Nuclear norm minimization MSE: %f\" % nnm_mse)\n",
        "\n",
        "softImpute_mse = ((X_filled_softimpute[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"SoftImpute MSE: %f\" % softImpute_mse)\n",
        "\n",
        "softImpute_no_biscale_mse = (\n",
        "    (X_filled_softimpute_no_biscale[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"SoftImpute without BiScale MSE: %f\" % softImpute_no_biscale_mse)\n",
        "\n",
        "\n",
        "knn_mse = ((X_filled_knn[missing_mask] - X[missing_mask]) ** 2).mean()\n",
        "print(\"knnImpute MSE: %f\" % knn_mse)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean squared element: 4.4201\n",
            "Imputing row 1/200 with 1 missing, elapsed time: 0.012\n",
            "Imputing row 101/200 with 0 missing, elapsed time: 0.018\n",
            "----------------------------------------------------------------------------\n",
            "\tSCS v2.1.0 - Splitting Conic Solver\n",
            "\t(c) Brendan O'Donoghue, Stanford University, 2012\n",
            "----------------------------------------------------------------------------\n",
            "Lin-sys: sparse-direct, nnz in A = 51532\n",
            "eps = 1.00e-05, alpha = 1.50, max_iters = 50000, normalize = 1, scale = 1.00\n",
            "acceleration_lookback = 10, rho_x = 1.00e-03\n",
            "Variables n = 32310, constraints m = 40310\n",
            "Cones:\tprimal zero / dual free vars: 4000\n",
            "\tlinear vars: 12000\n",
            "\tsd vars: 24310, sd blks: 1\n",
            "Setup time: 7.11e-02s\n",
            "----------------------------------------------------------------------------\n",
            " Iter | pri res | dua res | rel gap | pri obj | dua obj | kap/tau | time (s)\n",
            "----------------------------------------------------------------------------\n",
            "     0| 2.25e+21  2.08e+21  1.00e+00 -2.13e+24  4.35e+23  7.23e+23  4.12e-02 \n",
            "   100| 3.73e-04  4.40e-04  1.07e-05  2.53e+02  2.53e+02  7.39e-14  4.02e+00 \n",
            "   200| 2.33e-04  2.65e-04  1.24e-05  2.53e+02  2.53e+02  1.70e-13  8.00e+00 \n",
            "   300| 2.19e-04  4.34e-04  1.27e-05  2.53e+02  2.53e+02  8.01e-15  1.19e+01 \n",
            "   400| 4.58e-04  4.99e-04  3.80e-06  2.53e+02  2.53e+02  1.19e-14  1.58e+01 \n",
            "   500| 5.29e-05  4.33e-05  3.49e-05  2.53e+02  2.53e+02  8.75e-14  1.98e+01 \n",
            "   600| 4.04e-02  4.55e-02  1.50e-02  2.53e+02  2.45e+02  1.03e-13  2.37e+01 \n",
            "   700| 4.24e-04  6.40e-04  1.12e-04  2.53e+02  2.53e+02  5.50e-14  2.77e+01 \n",
            "   800| 2.77e-04  2.73e-04  1.23e-04  2.53e+02  2.53e+02  4.26e-14  3.16e+01 \n",
            "   900| 6.24e+20  4.79e+20  1.96e-03 -6.78e+22 -6.81e+22  3.13e+23  3.54e+01 \n",
            "  1000| 8.13e-05  1.08e-04  3.22e-05  2.53e+02  2.53e+02  6.11e-14  3.94e+01 \n",
            "  1100| 9.74e-05  1.48e-04  8.14e-05  2.53e+02  2.53e+02  3.22e-14  4.33e+01 \n",
            "  1200| 5.41e-04  7.74e-04  2.32e-05  2.53e+02  2.53e+02  3.21e-14  4.72e+01 \n",
            "  1300| 1.52e-04  2.17e-04  1.25e-04  2.53e+02  2.53e+02  5.26e-14  5.11e+01 \n",
            "  1400| 2.73e-03  4.35e-03  1.70e-03  2.53e+02  2.52e+02  6.31e-14  5.50e+01 \n",
            "  1500| 2.98e-03  4.83e-03  2.27e-03  2.53e+02  2.52e+02  6.88e-14  5.88e+01 \n",
            "  1600| 1.72e-04  2.45e-04  5.78e-05  2.53e+02  2.53e+02  9.62e-14  6.27e+01 \n",
            "  1700| 1.01e-04  1.52e-04  3.10e-05  2.53e+02  2.53e+02  8.39e-14  6.65e+01 \n",
            "  1800| 1.24e-03  1.96e-03  8.73e-04  2.53e+02  2.52e+02  4.30e-14  7.04e+01 \n",
            "  1900| 4.84e-05  7.53e-05  1.26e-05  2.53e+02  2.53e+02  1.49e-14  7.42e+01 \n",
            "  2000| 5.07e-05  7.64e-05  1.70e-05  2.53e+02  2.53e+02  1.22e-14  7.80e+01 \n",
            "  2100| 6.29e-05  9.86e-05  1.94e-05  2.53e+02  2.53e+02  2.51e-14  8.19e+01 \n",
            "  2200| 7.05e-05  9.76e-05  1.51e-05  2.53e+02  2.53e+02  9.78e-14  8.57e+01 \n",
            "  2300| 3.25e-05  4.13e-05  7.41e-09  2.53e+02  2.53e+02  3.31e-14  8.96e+01 \n",
            "  2400| 9.77e-05  1.50e-04  1.82e-05  2.53e+02  2.53e+02  4.80e-14  9.34e+01 \n",
            "  2500| 7.95e-05  1.12e-04  2.27e-05  2.53e+02  2.53e+02  1.16e-14  9.72e+01 \n",
            "  2600| 1.32e-05  1.78e-05  5.14e-07  2.53e+02  2.53e+02  6.90e-14  1.01e+02 \n",
            "  2700| 2.27e-05  2.71e-05  1.45e-05  2.53e+02  2.53e+02  1.22e-13  1.05e+02 \n",
            "  2800| 1.56e-05  1.98e-05  1.04e-05  2.53e+02  2.53e+02  9.76e-15  1.09e+02 \n",
            "  2900| 2.15e-05  3.14e-05  2.72e-05  2.53e+02  2.53e+02  3.89e-14  1.13e+02 \n",
            "  3000| 1.57e-05  1.93e-05  7.53e-06  2.53e+02  2.53e+02  6.25e-14  1.16e+02 \n",
            "  3100| 2.06e-05  2.14e-05  9.34e-06  2.53e+02  2.53e+02  2.43e-15  1.20e+02 \n",
            "  3180| 8.66e-06  7.91e-06  4.37e-06  2.53e+02  2.53e+02  7.64e-14  1.23e+02 \n",
            "----------------------------------------------------------------------------\n",
            "Status: Solved\n",
            "Timing: Solve time: 1.23e+02s\n",
            "\tLin-sys: nnz in L factor: 127763, avg solve time: 1.11e-03s\n",
            "\tCones: avg projection time: 2.45e-02s\n",
            "\tAcceleration: avg step time: 1.10e-02s\n",
            "----------------------------------------------------------------------------\n",
            "Error metrics:\n",
            "dist(s, K) = 1.7659e-09, dist(y, K*) = 1.8476e-09, s'y/|s||y| = -5.6933e-13\n",
            "primal res: |Ax + s - b|_2 / (1 + |b|_2) = 8.6606e-06\n",
            "dual res:   |A'y + c|_2 / (1 + |c|_2) = 7.9144e-06\n",
            "rel gap:    |c'x + b'y| / (1 + |c'x| + |b'y|) = 4.3702e-06\n",
            "----------------------------------------------------------------------------\n",
            "c'x = 252.9263, -b'y = 252.9286\n",
            "============================================================================\n",
            "[BiScaler] Initial log residual value = 6.224517\n",
            "[BiScaler] Iter 1: log residual = 1.005431, log improvement ratio=5.219086\n",
            "[BiScaler] Iter 2: log residual = -0.819173, log improvement ratio=1.824604\n",
            "[BiScaler] Iter 3: log residual = -2.580621, log improvement ratio=1.761448\n",
            "[BiScaler] Iter 4: log residual = -4.321559, log improvement ratio=1.740937\n",
            "[BiScaler] Iter 5: log residual = -6.056305, log improvement ratio=1.734746\n",
            "[BiScaler] Iter 6: log residual = -7.788422, log improvement ratio=1.732117\n",
            "[BiScaler] Iter 7: log residual = -9.519164, log improvement ratio=1.730742\n",
            "[BiScaler] Iter 8: log residual = -11.249132, log improvement ratio=1.729968\n",
            "[BiScaler] Iter 9: log residual = -12.978651, log improvement ratio=1.729520\n",
            "[BiScaler] Iter 10: log residual = -14.707908, log improvement ratio=1.729256\n",
            "[BiScaler] Iter 11: log residual = -16.437006, log improvement ratio=1.729098\n",
            "[BiScaler] Iter 12: log residual = -18.166007, log improvement ratio=1.729001\n",
            "[BiScaler] Iter 13: log residual = -19.894948, log improvement ratio=1.728940\n",
            "[BiScaler] Iter 14: log residual = -21.623848, log improvement ratio=1.728900\n",
            "[BiScaler] Iter 15: log residual = -23.352722, log improvement ratio=1.728873\n",
            "[BiScaler] Iter 16: log residual = -25.081576, log improvement ratio=1.728855\n",
            "[BiScaler] Iter 17: log residual = -26.810417, log improvement ratio=1.728841\n",
            "[BiScaler] Iter 18: log residual = -28.539247, log improvement ratio=1.728830\n",
            "[BiScaler] Iter 19: log residual = -30.268069, log improvement ratio=1.728822\n",
            "[BiScaler] Iter 20: log residual = -31.996885, log improvement ratio=1.728816\n",
            "[BiScaler] Iter 21: log residual = -33.725695, log improvement ratio=1.728810\n",
            "[BiScaler] Iter 22: log residual = -35.454501, log improvement ratio=1.728806\n",
            "[BiScaler] Iter 23: log residual = -37.183304, log improvement ratio=1.728803\n",
            "[BiScaler] Iter 24: log residual = -38.912103, log improvement ratio=1.728799\n",
            "[BiScaler] Iter 25: log residual = -40.640901, log improvement ratio=1.728798\n",
            "[BiScaler] Iter 26: log residual = -42.369694, log improvement ratio=1.728792\n",
            "[BiScaler] Iter 27: log residual = -44.098486, log improvement ratio=1.728793\n",
            "[BiScaler] Iter 28: log residual = -45.827291, log improvement ratio=1.728804\n",
            "[BiScaler] Iter 29: log residual = -47.556084, log improvement ratio=1.728793\n",
            "[BiScaler] Iter 30: log residual = -49.284913, log improvement ratio=1.728830\n",
            "[BiScaler] Iter 31: log residual = -51.013544, log improvement ratio=1.728631\n",
            "[BiScaler] Iter 32: log residual = -52.742112, log improvement ratio=1.728568\n",
            "[BiScaler] Iter 33: log residual = -54.471719, log improvement ratio=1.729607\n",
            "[BiScaler] Iter 34: log residual = -56.200049, log improvement ratio=1.728330\n",
            "[BiScaler] Iter 35: log residual = -57.922210, log improvement ratio=1.722161\n",
            "[BiScaler] Iter 36: log residual = -59.661294, log improvement ratio=1.739083\n",
            "[BiScaler] Iter 37: log residual = -61.383860, log improvement ratio=1.722567\n",
            "[BiScaler] Iter 38: log residual = -63.036593, log improvement ratio=1.652733\n",
            "[BiScaler] Iter 39: log residual = -64.541409, log improvement ratio=1.504817\n",
            "[BiScaler] Iter 40: log residual = -65.607341, log improvement ratio=1.065932\n",
            "[BiScaler] Iter 41: log residual = -66.106846, log improvement ratio=0.499505\n",
            "[BiScaler] Iter 42: log residual = -66.279618, log improvement ratio=0.172772\n",
            "[BiScaler] Iter 43: log residual = -66.227398, log improvement ratio=-0.052220\n",
            "[SoftImpute] Max Singular Value of X_init = 34.587047\n",
            "[SoftImpute] Iter 1: observed MAE=0.031801 rank=19\n",
            "[SoftImpute] Iter 2: observed MAE=0.032585 rank=19\n",
            "[SoftImpute] Iter 3: observed MAE=0.033564 rank=19\n",
            "[SoftImpute] Iter 4: observed MAE=0.033870 rank=20\n",
            "[SoftImpute] Iter 5: observed MAE=0.033999 rank=20\n",
            "[SoftImpute] Iter 6: observed MAE=0.034135 rank=20\n",
            "[SoftImpute] Iter 7: observed MAE=0.034276 rank=20\n",
            "[SoftImpute] Iter 8: observed MAE=0.034429 rank=20\n",
            "[SoftImpute] Iter 9: observed MAE=0.034601 rank=20\n",
            "[SoftImpute] Iter 10: observed MAE=0.034623 rank=19\n",
            "[SoftImpute] Iter 11: observed MAE=0.034428 rank=18\n",
            "[SoftImpute] Iter 12: observed MAE=0.033699 rank=16\n",
            "[SoftImpute] Iter 13: observed MAE=0.032385 rank=13\n",
            "[SoftImpute] Iter 14: observed MAE=0.030570 rank=10\n",
            "[SoftImpute] Iter 15: observed MAE=0.028901 rank=9\n",
            "[SoftImpute] Iter 16: observed MAE=0.027747 rank=7\n",
            "[SoftImpute] Iter 17: observed MAE=0.026814 rank=7\n",
            "[SoftImpute] Iter 18: observed MAE=0.026224 rank=6\n",
            "[SoftImpute] Iter 19: observed MAE=0.025746 rank=6\n",
            "[SoftImpute] Iter 20: observed MAE=0.025512 rank=6\n",
            "[SoftImpute] Iter 21: observed MAE=0.025393 rank=6\n",
            "[SoftImpute] Iter 22: observed MAE=0.025330 rank=6\n",
            "[SoftImpute] Iter 23: observed MAE=0.025294 rank=6\n",
            "[SoftImpute] Stopped after iteration 23 for lambda=0.691741\n",
            "[SoftImpute] Max Singular Value of X_init = 79.573258\n",
            "[SoftImpute] Iter 1: observed MAE=0.067800 rank=20\n",
            "[SoftImpute] Iter 2: observed MAE=0.068429 rank=20\n",
            "[SoftImpute] Iter 3: observed MAE=0.068544 rank=19\n",
            "[SoftImpute] Iter 4: observed MAE=0.068220 rank=18\n",
            "[SoftImpute] Iter 5: observed MAE=0.066699 rank=16\n",
            "[SoftImpute] Iter 6: observed MAE=0.065307 rank=15\n",
            "[SoftImpute] Iter 7: observed MAE=0.062441 rank=13\n",
            "[SoftImpute] Iter 8: observed MAE=0.060135 rank=13\n",
            "[SoftImpute] Iter 9: observed MAE=0.058110 rank=11\n",
            "[SoftImpute] Iter 10: observed MAE=0.055489 rank=9\n",
            "[SoftImpute] Iter 11: observed MAE=0.053092 rank=9\n",
            "[SoftImpute] Iter 12: observed MAE=0.051551 rank=8\n",
            "[SoftImpute] Iter 13: observed MAE=0.050578 rank=8\n",
            "[SoftImpute] Iter 14: observed MAE=0.050089 rank=8\n",
            "[SoftImpute] Iter 15: observed MAE=0.049369 rank=6\n",
            "[SoftImpute] Iter 16: observed MAE=0.047804 rank=6\n",
            "[SoftImpute] Iter 17: observed MAE=0.046271 rank=5\n",
            "[SoftImpute] Iter 18: observed MAE=0.045334 rank=5\n",
            "[SoftImpute] Iter 19: observed MAE=0.044782 rank=5\n",
            "[SoftImpute] Iter 20: observed MAE=0.044441 rank=5\n",
            "[SoftImpute] Iter 21: observed MAE=0.043700 rank=4\n",
            "[SoftImpute] Iter 22: observed MAE=0.043283 rank=4\n",
            "[SoftImpute] Iter 23: observed MAE=0.043059 rank=4\n",
            "[SoftImpute] Iter 24: observed MAE=0.042935 rank=4\n",
            "[SoftImpute] Iter 25: observed MAE=0.042861 rank=4\n",
            "[SoftImpute] Iter 26: observed MAE=0.042817 rank=4\n",
            "[SoftImpute] Stopped after iteration 26 for lambda=1.591465\n",
            "meanFill MSE: 4.965477\n",
            "Iterative Imputer norm minimization MSE: 0.000000\n",
            "Nuclear norm minimization MSE: 0.000000\n",
            "SoftImpute MSE: 0.013702\n",
            "SoftImpute without BiScale MSE: 0.011530\n",
            "knnImpute MSE: 0.640178\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJQp7i3FCroZ",
        "colab_type": "text"
      },
      "source": [
        "### RESULTS\n",
        "We see that the Iterative Imputer and the Nuclear norm minimuxation has an MSE of 0.\n",
        "\n",
        "We see that simply filling missing values with the mean (meanFill) results in exponentially larger error. \n",
        "\n",
        "For the love of all that is good in the world, use iterative imputation over simple methods (where appropriate)"
      ]
    }
  ]
}