User Improvement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Individual User Improvement Metrics\n",
    "\n",
    "> Groups entries made by users together and orders them from oldest to newest. This can facillitate future research exploring how individual citizen scientists improve over time.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading Data Procedure:\n",
    "\n",
    "## Use:\n",
    "Run the following cell if you would like to get data from the GLOBE API\n",
    "\n",
    "## Procedure\n",
    "Downloading data from the API follows these steps:\n",
    "- Request the non GEOJSON data from the GLOBE API\n",
    "- Get the results from the JSON and pass it into a pandas dataframe\n",
    "- Unpack the \"data\" entry\n",
    "- Merge the data onto the original dataframe\n",
    "- Remove the \"data\" entry from the original dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>measuredDate</th>\n",
       "      <th>createDate</th>\n",
       "      <th>updateDate</th>\n",
       "      <th>publishDate</th>\n",
       "      <th>organizationId</th>\n",
       "      <th>organizationName</th>\n",
       "      <th>siteId</th>\n",
       "      <th>siteName</th>\n",
       "      <th>countryName</th>\n",
       "      <th>...</th>\n",
       "      <th>mosquitohabitatmapperComments</th>\n",
       "      <th>mosquitohabitatmapperMosquitoPupae</th>\n",
       "      <th>mosquitohabitatmapperWaterSourcePhotoUrls</th>\n",
       "      <th>mosquitohabitatmapperDataSource</th>\n",
       "      <th>mosquitohabitatmapperLarvaFullBodyPhotoUrls</th>\n",
       "      <th>mosquitohabitatmapperMeasurementLatitude</th>\n",
       "      <th>mosquitohabitatmapperLastIdentifyStage</th>\n",
       "      <th>mosquitohabitatmapperWaterSourceType</th>\n",
       "      <th>mosquitohabitatmapperMosquitoHabitatMapperId</th>\n",
       "      <th>mosquitohabitatmapperMeasurementLongitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2018-11-25</td>\n",
       "      <td>2020-01-25T18:09:52</td>\n",
       "      <td>2020-01-25T18:09:52</td>\n",
       "      <td>2020-02-14T20:29:11</td>\n",
       "      <td>13063641.0</td>\n",
       "      <td>GPM Satellite Mission</td>\n",
       "      <td>35785</td>\n",
       "      <td>18SUJ105472</td>\n",
       "      <td>United States</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2018/11/2...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>39.2538</td>\n",
       "      <td>None</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>5188</td>\n",
       "      <td>-77.1959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2019-04-07</td>\n",
       "      <td>2020-01-25T18:24:27</td>\n",
       "      <td>2020-01-25T18:24:27</td>\n",
       "      <td>2020-03-20T22:19:48</td>\n",
       "      <td>13063641.0</td>\n",
       "      <td>GPM Satellite Mission</td>\n",
       "      <td>35785</td>\n",
       "      <td>18SUJ105472</td>\n",
       "      <td>United States</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2019/04/0...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>39.2535</td>\n",
       "      <td>identify</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>10365</td>\n",
       "      <td>-77.1960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2019-04-07</td>\n",
       "      <td>2020-01-25T18:24:27</td>\n",
       "      <td>2020-01-25T18:24:27</td>\n",
       "      <td>2020-03-20T22:19:48</td>\n",
       "      <td>13063641.0</td>\n",
       "      <td>GPM Satellite Mission</td>\n",
       "      <td>35785</td>\n",
       "      <td>18SUJ105472</td>\n",
       "      <td>United States</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2019/04/0...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>39.2536</td>\n",
       "      <td>identify</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>10360</td>\n",
       "      <td>-77.1956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2019-05-29</td>\n",
       "      <td>2020-01-25T18:29:36</td>\n",
       "      <td>2020-01-25T18:29:36</td>\n",
       "      <td>2020-03-20T22:19:48</td>\n",
       "      <td>13063641.0</td>\n",
       "      <td>GPM Satellite Mission</td>\n",
       "      <td>35785</td>\n",
       "      <td>18SUJ105472</td>\n",
       "      <td>United States</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2019/05/2...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>https://data.globe.gov/system/photos/2019/05/2...</td>\n",
       "      <td>39.2542</td>\n",
       "      <td>identify-siphon-pecten</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>12424</td>\n",
       "      <td>-77.1962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2019-08-04</td>\n",
       "      <td>2020-01-25T18:45:20</td>\n",
       "      <td>2020-01-25T18:45:20</td>\n",
       "      <td>2020-03-20T22:19:48</td>\n",
       "      <td>13063641.0</td>\n",
       "      <td>GPM Satellite Mission</td>\n",
       "      <td>35785</td>\n",
       "      <td>18SUJ105472</td>\n",
       "      <td>United States</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2019/08/0...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>39.2536</td>\n",
       "      <td>identify</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>14822</td>\n",
       "      <td>-77.1957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22452</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2020-05-30</td>\n",
       "      <td>2020-05-30T18:25:03</td>\n",
       "      <td>2020-05-30T19:30:02</td>\n",
       "      <td>2020-11-23T21:32:54</td>\n",
       "      <td>14054356.0</td>\n",
       "      <td>lycee Thilmakha</td>\n",
       "      <td>200580</td>\n",
       "      <td>28PCB553673</td>\n",
       "      <td>Senegal</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>https://data.globe.gov/system/photos/2020/05/3...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>https://data.globe.gov/system/photos/2020/05/3...</td>\n",
       "      <td>15.0778</td>\n",
       "      <td>identify-siphon-shape</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>22772</td>\n",
       "      <td>-16.3463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22453</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2020-05-29</td>\n",
       "      <td>2020-06-03T07:30:03</td>\n",
       "      <td>2020-06-03T16:15:20</td>\n",
       "      <td>2020-11-23T21:32:54</td>\n",
       "      <td>19841715.0</td>\n",
       "      <td>Madagascar Citizen Science</td>\n",
       "      <td>201123</td>\n",
       "      <td>38KQE645064</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>https://data.globe.gov/system/photos/2020/05/2...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>-18.9168</td>\n",
       "      <td>identify</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>22806</td>\n",
       "      <td>47.5121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22454</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2020-05-06</td>\n",
       "      <td>2020-07-14T08:40:05</td>\n",
       "      <td>2020-07-14T13:40:07</td>\n",
       "      <td>2020-11-23T21:32:54</td>\n",
       "      <td>18306968.0</td>\n",
       "      <td>Taiwan Partnership Citizen Science</td>\n",
       "      <td>208771</td>\n",
       "      <td>51RUH399684</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>https://data.globe.gov/system/photos/2020/05/0...</td>\n",
       "      <td>25.0235</td>\n",
       "      <td>identify-basal-tuft</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>24654</td>\n",
       "      <td>121.4134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22455</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2020-05-31</td>\n",
       "      <td>2020-07-18T23:35:02</td>\n",
       "      <td>2020-07-18T23:35:02</td>\n",
       "      <td>2020-11-23T21:32:54</td>\n",
       "      <td>14054356.0</td>\n",
       "      <td>lycee Thilmakha</td>\n",
       "      <td>209660</td>\n",
       "      <td>28PDB055429</td>\n",
       "      <td>Senegal</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>https://data.globe.gov/system/photos/2020/05/3...</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>14.8591</td>\n",
       "      <td>identify-aedes-tuft</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>24880</td>\n",
       "      <td>-15.8784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22456</th>\n",
       "      <td>mosquito_habitat_mapper</td>\n",
       "      <td>2020-01-29</td>\n",
       "      <td>2020-07-23T07:13:04</td>\n",
       "      <td>2020-07-23T07:13:04</td>\n",
       "      <td>2020-11-23T21:32:54</td>\n",
       "      <td>217639.0</td>\n",
       "      <td>Suranaree University of Technology</td>\n",
       "      <td>210372</td>\n",
       "      <td>47PNK984553</td>\n",
       "      <td>Thailand</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "      <td>GLOBE Observer App</td>\n",
       "      <td>None</td>\n",
       "      <td>8.6420</td>\n",
       "      <td>identify-photo-prep</td>\n",
       "      <td>container: artificial</td>\n",
       "      <td>25049</td>\n",
       "      <td>99.8951</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22457 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      protocol measuredDate           createDate  \\\n",
       "0      mosquito_habitat_mapper   2018-11-25  2020-01-25T18:09:52   \n",
       "1      mosquito_habitat_mapper   2019-04-07  2020-01-25T18:24:27   \n",
       "2      mosquito_habitat_mapper   2019-04-07  2020-01-25T18:24:27   \n",
       "3      mosquito_habitat_mapper   2019-05-29  2020-01-25T18:29:36   \n",
       "4      mosquito_habitat_mapper   2019-08-04  2020-01-25T18:45:20   \n",
       "...                        ...          ...                  ...   \n",
       "22452  mosquito_habitat_mapper   2020-05-30  2020-05-30T18:25:03   \n",
       "22453  mosquito_habitat_mapper   2020-05-29  2020-06-03T07:30:03   \n",
       "22454  mosquito_habitat_mapper   2020-05-06  2020-07-14T08:40:05   \n",
       "22455  mosquito_habitat_mapper   2020-05-31  2020-07-18T23:35:02   \n",
       "22456  mosquito_habitat_mapper   2020-01-29  2020-07-23T07:13:04   \n",
       "\n",
       "                updateDate          publishDate  organizationId  \\\n",
       "0      2020-01-25T18:09:52  2020-02-14T20:29:11      13063641.0   \n",
       "1      2020-01-25T18:24:27  2020-03-20T22:19:48      13063641.0   \n",
       "2      2020-01-25T18:24:27  2020-03-20T22:19:48      13063641.0   \n",
       "3      2020-01-25T18:29:36  2020-03-20T22:19:48      13063641.0   \n",
       "4      2020-01-25T18:45:20  2020-03-20T22:19:48      13063641.0   \n",
       "...                    ...                  ...             ...   \n",
       "22452  2020-05-30T19:30:02  2020-11-23T21:32:54      14054356.0   \n",
       "22453  2020-06-03T16:15:20  2020-11-23T21:32:54      19841715.0   \n",
       "22454  2020-07-14T13:40:07  2020-11-23T21:32:54      18306968.0   \n",
       "22455  2020-07-18T23:35:02  2020-11-23T21:32:54      14054356.0   \n",
       "22456  2020-07-23T07:13:04  2020-11-23T21:32:54        217639.0   \n",
       "\n",
       "                         organizationName  siteId     siteName    countryName  \\\n",
       "0                   GPM Satellite Mission   35785  18SUJ105472  United States   \n",
       "1                   GPM Satellite Mission   35785  18SUJ105472  United States   \n",
       "2                   GPM Satellite Mission   35785  18SUJ105472  United States   \n",
       "3                   GPM Satellite Mission   35785  18SUJ105472  United States   \n",
       "4                   GPM Satellite Mission   35785  18SUJ105472  United States   \n",
       "...                                   ...     ...          ...            ...   \n",
       "22452                     lycee Thilmakha  200580  28PCB553673        Senegal   \n",
       "22453          Madagascar Citizen Science  201123  38KQE645064           None   \n",
       "22454  Taiwan Partnership Citizen Science  208771  51RUH399684           None   \n",
       "22455                     lycee Thilmakha  209660  28PDB055429        Senegal   \n",
       "22456  Suranaree University of Technology  210372  47PNK984553       Thailand   \n",
       "\n",
       "       ... mosquitohabitatmapperComments  mosquitohabitatmapperMosquitoPupae  \\\n",
       "0      ...                          None                               False   \n",
       "1      ...                          None                               False   \n",
       "2      ...                          None                               False   \n",
       "3      ...                          None                               False   \n",
       "4      ...                          None                               False   \n",
       "...    ...                           ...                                 ...   \n",
       "22452  ...                          None                                True   \n",
       "22453  ...                          None                               False   \n",
       "22454  ...                          None                                True   \n",
       "22455  ...                          None                                True   \n",
       "22456  ...                          None                               False   \n",
       "\n",
       "               mosquitohabitatmapperWaterSourcePhotoUrls  \\\n",
       "0      https://data.globe.gov/system/photos/2018/11/2...   \n",
       "1      https://data.globe.gov/system/photos/2019/04/0...   \n",
       "2      https://data.globe.gov/system/photos/2019/04/0...   \n",
       "3      https://data.globe.gov/system/photos/2019/05/2...   \n",
       "4      https://data.globe.gov/system/photos/2019/08/0...   \n",
       "...                                                  ...   \n",
       "22452  https://data.globe.gov/system/photos/2020/05/3...   \n",
       "22453  https://data.globe.gov/system/photos/2020/05/2...   \n",
       "22454                                               None   \n",
       "22455  https://data.globe.gov/system/photos/2020/05/3...   \n",
       "22456                                               None   \n",
       "\n",
       "       mosquitohabitatmapperDataSource  \\\n",
       "0                   GLOBE Observer App   \n",
       "1                   GLOBE Observer App   \n",
       "2                   GLOBE Observer App   \n",
       "3                   GLOBE Observer App   \n",
       "4                   GLOBE Observer App   \n",
       "...                                ...   \n",
       "22452               GLOBE Observer App   \n",
       "22453               GLOBE Observer App   \n",
       "22454               GLOBE Observer App   \n",
       "22455               GLOBE Observer App   \n",
       "22456               GLOBE Observer App   \n",
       "\n",
       "             mosquitohabitatmapperLarvaFullBodyPhotoUrls  \\\n",
       "0                                                   None   \n",
       "1                                                   None   \n",
       "2                                                   None   \n",
       "3      https://data.globe.gov/system/photos/2019/05/2...   \n",
       "4                                                   None   \n",
       "...                                                  ...   \n",
       "22452  https://data.globe.gov/system/photos/2020/05/3...   \n",
       "22453                                               None   \n",
       "22454  https://data.globe.gov/system/photos/2020/05/0...   \n",
       "22455                                               None   \n",
       "22456                                               None   \n",
       "\n",
       "      mosquitohabitatmapperMeasurementLatitude  \\\n",
       "0                                      39.2538   \n",
       "1                                      39.2535   \n",
       "2                                      39.2536   \n",
       "3                                      39.2542   \n",
       "4                                      39.2536   \n",
       "...                                        ...   \n",
       "22452                                  15.0778   \n",
       "22453                                 -18.9168   \n",
       "22454                                  25.0235   \n",
       "22455                                  14.8591   \n",
       "22456                                   8.6420   \n",
       "\n",
       "      mosquitohabitatmapperLastIdentifyStage  \\\n",
       "0                                       None   \n",
       "1                                   identify   \n",
       "2                                   identify   \n",
       "3                     identify-siphon-pecten   \n",
       "4                                   identify   \n",
       "...                                      ...   \n",
       "22452                  identify-siphon-shape   \n",
       "22453                               identify   \n",
       "22454                    identify-basal-tuft   \n",
       "22455                    identify-aedes-tuft   \n",
       "22456                    identify-photo-prep   \n",
       "\n",
       "      mosquitohabitatmapperWaterSourceType  \\\n",
       "0                    container: artificial   \n",
       "1                    container: artificial   \n",
       "2                    container: artificial   \n",
       "3                    container: artificial   \n",
       "4                    container: artificial   \n",
       "...                                    ...   \n",
       "22452                container: artificial   \n",
       "22453                container: artificial   \n",
       "22454                container: artificial   \n",
       "22455                container: artificial   \n",
       "22456                container: artificial   \n",
       "\n",
       "      mosquitohabitatmapperMosquitoHabitatMapperId  \\\n",
       "0                                             5188   \n",
       "1                                            10365   \n",
       "2                                            10360   \n",
       "3                                            12424   \n",
       "4                                            14822   \n",
       "...                                            ...   \n",
       "22452                                        22772   \n",
       "22453                                        22806   \n",
       "22454                                        24654   \n",
       "22455                                        24880   \n",
       "22456                                        25049   \n",
       "\n",
       "       mosquitohabitatmapperMeasurementLongitude  \n",
       "0                                       -77.1959  \n",
       "1                                       -77.1960  \n",
       "2                                       -77.1956  \n",
       "3                                       -77.1962  \n",
       "4                                       -77.1957  \n",
       "...                                          ...  \n",
       "22452                                   -16.3463  \n",
       "22453                                    47.5121  \n",
       "22454                                   121.4134  \n",
       "22455                                   -15.8784  \n",
       "22456                                    99.8951  \n",
       "\n",
       "[22457 rows x 38 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "start_date = \"2017-05-29\"\n",
    "end_date = \"2020-05-31\"\n",
    "url = f\"https://api.globe.gov/search/v1/measurement/protocol/measureddate/?protocols=mosquito_habitat_mapper&startdate={start_date}&enddate={end_date}&geojson=FALSE&sample=FALSE\"\n",
    "\n",
    "# downloads data from the GLOBE API\n",
    "response = requests.get(url)\n",
    "results = response.json()[\"results\"]\n",
    "\n",
    "df = pd.DataFrame(results)\n",
    "\n",
    "#Expand the 'data' column by listing the contents and passing as a new dataframe\n",
    "df = pd.concat([df, pd.DataFrame(list(df['data']))], axis=1)\n",
    "\n",
    "#Drop the previously nested data column\n",
    "df = df.drop('data', 1)\n",
    "\n",
    "# display the dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting/Grouping Procedure:\n",
    "\n",
    "- Groups entries with the same User ID\n",
    "- Creates a list of the UserID's and the size of the groups\n",
    "- Sorts the list by size of group in descending order\n",
    "- Iterates through the list of sorted groups and takes each group, sorts the\n",
    "observations by User ID, and then adds it to the final data frame\n",
    "\n",
    "Result: a observations with entries that share the same user being grouped\n",
    "into the same part of the spreadsheet. These groups also show the observations \n",
    "made by the user from oldest to latest. This will allow people to analyze the\n",
    "improvement of certain users over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creates groups by similar userID\n",
    "groups = df.groupby(\"mosquitohabitatmapperUserid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mosquitohabitatmapperUserid</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51045191</td>\n",
       "      <td>739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50985322</td>\n",
       "      <td>565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5284745</td>\n",
       "      <td>456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>51046601</td>\n",
       "      <td>408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21748177</td>\n",
       "      <td>374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4899</th>\n",
       "      <td>52560018</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4900</th>\n",
       "      <td>52559362</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4901</th>\n",
       "      <td>52559352</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4902</th>\n",
       "      <td>52559112</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4903</th>\n",
       "      <td>11086</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4904 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mosquitohabitatmapperUserid    0\n",
       "0                        51045191  739\n",
       "1                        50985322  565\n",
       "2                         5284745  456\n",
       "3                        51046601  408\n",
       "4                        21748177  374\n",
       "...                           ...  ...\n",
       "4899                     52560018    1\n",
       "4900                     52559362    1\n",
       "4901                     52559352    1\n",
       "4902                     52559112    1\n",
       "4903                        11086    1\n",
       "\n",
       "[4904 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creates a list of users (sorted by amt of entries)\n",
    "gb = groups.size()\n",
    "gb = gb.sort_values(ascending = False)\n",
    "gb = gb.reset_index()\n",
    "gb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-8df48c1dfa85>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mfinal_df\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0muserID\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mgb\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"mosquitohabitatmapperUserid\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m     \u001b[0mtemp_df\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgroups\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_group\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0muserID\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msort_values\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"measuredDate\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m     \u001b[0mfinal_df\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfinal_df\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtemp_df\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\frame.py\u001b[0m in \u001b[0;36msort_values\u001b[1;34m(self, by, axis, ascending, inplace, kind, na_position, ignore_index)\u001b[0m\n\u001b[0;32m   4931\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   4932\u001b[0m             indexer = nargsort(\n\u001b[1;32m-> 4933\u001b[1;33m                 \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkind\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mkind\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mascending\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mascending\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mna_position\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mna_position\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   4934\u001b[0m             )\n\u001b[0;32m   4935\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\sorting.py\u001b[0m in \u001b[0;36mnargsort\u001b[1;34m(items, kind, ascending, na_position)\u001b[0m\n\u001b[0;32m    258\u001b[0m     \"\"\"\n\u001b[0;32m    259\u001b[0m     \u001b[0mitems\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mextract_array\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 260\u001b[1;33m     \u001b[0mmask\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0misna\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    261\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    262\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mis_extension_array_dtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\dtypes\\missing.py\u001b[0m in \u001b[0;36misna\u001b[1;34m(obj)\u001b[0m\n\u001b[0;32m    124\u001b[0m     \u001b[0mName\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    125\u001b[0m     \"\"\"\n\u001b[1;32m--> 126\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0m_isna\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    127\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    128\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\dtypes\\missing.py\u001b[0m in \u001b[0;36m_isna_new\u001b[1;34m(obj)\u001b[0m\n\u001b[0;32m    150\u001b[0m         ),\n\u001b[0;32m    151\u001b[0m     ):\n\u001b[1;32m--> 152\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0m_isna_ndarraylike\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    153\u001b[0m     \u001b[1;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mABCGeneric\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    154\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mobj\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_constructor\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misna\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0misna\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\dtypes\\missing.py\u001b[0m in \u001b[0;36m_isna_ndarraylike\u001b[1;34m(obj)\u001b[0m\n\u001b[0;32m    254\u001b[0m             \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mempty\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbool\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    255\u001b[0m             \u001b[0mvec\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlibmissing\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misnaobj\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 256\u001b[1;33m             \u001b[0mresult\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m...\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvec\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    257\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    258\u001b[0m     \u001b[1;32melif\u001b[0m \u001b[0mneeds_i8_conversion\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# for each of the userID's, it sorts their entries (oldest - newest) and adds to master dataframe\n",
    "final_df = pd.DataFrame()\n",
    "for userID in gb[\"mosquitohabitatmapperUserid\"]:\n",
    "    temp_df = groups.get_group(userID).sort_values(\"measuredDate\")\n",
    "    final_df = final_df.append(temp_df, ignore_index = True)\n",
    "\n",
    "final_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# outputs to file\n",
    "final_df.to_csv(\"User_List.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}