notebooks/03.5-rainfall.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59ac1766-15cc-4287-bb29-7020b58e9c33",
   "metadata": {},
   "source": [
    "# Brooklyn Weather\n",
    "\n",
    "This simple notebook gets data from [NOAA](https://www.ncdc.noaa.gov/cdo-web/datasets/GHCND/stations/GHCND:US1NYKN0025/detail) for daily percipitation.\n",
    "\n",
    "Not doing any analysis, just want to plot it for context.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91939bb2-3761-479a-bb21-a3c99dc900aa",
   "metadata": {},
   "source": [
    "# Rainfall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "855a9892-38d5-4a96-98d4-a453f3eb50df",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df = pd.read_csv('../data/raw/weather/brooklyn-2021.csv', parse_dates=['DATE'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9860ed96-5a09-484e-a84f-78f47174a7f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf61db0-4996-477b-9c41-c72a54611ba5",
   "metadata": {},
   "source": [
    "I expected this data set to have 365 entries.  I'm not using it for any detailed analysis, just the basics for situational awareness, so I’m not sure I care. However, I am curious, so maybe just a quick look-see at what is missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2cba23-7423-4064-9529-7e187f272775",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df.NAME.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ea3ac1b-2dd0-4b0d-b393-f49501a6c05d",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df['DATE'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c72da7e-b05a-451d-b8af-18ca135c855b",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df['DATE'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe5fd39-ee46-48ac-8114-633c8fd67428",
   "metadata": {},
   "source": [
    "Let's see which dates are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4d547f4-0acd-435b-a013-f1e1d7681e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_days_2021 = set(pd.date_range(start='1/1/2021', end='12/31/2021').tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "538166e6-1312-4405-bb33-529668933660",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(all_days_2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1adce687-4ed9-48c2-8647-76b9861516b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "days_we_have = set(rainfall_df['DATE'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00717f17-ecae-4eef-9447-c0b99c3a17dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(days_we_have)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd313dc2-7d28-4da1-8f9a-76470fb1a98c",
   "metadata": {},
   "source": [
    "**Note:** This is the size of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9ac465a-9ff8-40b7-ae80-2689d14f1c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_days = all_days_2021 - days_we_have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0581ac98-bd35-4835-aec8-923e5508c0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(missing_dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ebad01e-fa6a-497d-baf5-93218559e8d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ + len(rainfall_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e010446f-b4bc-43d6-997c-ac1e1f19ffe1",
   "metadata": {},
   "source": [
    "OK, goofy test number one passes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0243b23-186b-4783-9b57-c42721027804",
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_days"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f88122-0c2c-4b5f-8edd-94ee40125fbe",
   "metadata": {},
   "source": [
    "A big chunck of July is missing.  For a **real** analysis of rainfall that might be an issue?\n",
    "\n",
    "I should care, but ... meh.\n",
    "\n",
    "Another issue from the dataframe info is 5 of the rows have null PRCP values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8864f8a-6787-4664-a90e-4366e415f8ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df['PRCP'].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee81f296-80f6-4811-bdc4-f18fce936124",
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_df = rainfall_df[rainfall_df['PRCP'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6ece0ca-3d79-44bc-a3c1-9ba1201ade3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd250e63-ba54-4c09-8dd9-907423c8621c",
   "metadata": {},
   "source": [
    "They're all in February."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f195813a-6398-4e96-945e-ecc6588fb246",
   "metadata": {},
   "source": [
    "Let's check out the rainfall information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d6be6d-ce0c-4183-97e8-bcbbb7823532",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "rainfall_df['PRCP'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b1e0da5-a628-4d27-9f64-bb64e8d2bb22",
   "metadata": {},
   "source": [
    "Even though we're missing a big chunck of July this is still a lot of rain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9badaa7f-4ebb-4423-b8f5-359afcefc581",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts = pd.Series(rainfall_df['PRCP'].values, index=rainfall_df['DATE'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bfa5b44-00df-4d0d-9852-996d5fe129af",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts.plot(figsize=(20, 10));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a549c85f-cd96-48ea-8001-2477828e6d82",
   "metadata": {},
   "source": [
    "In keeping with my workflow save this into the processed/brooklyn directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a02bc616-b3ba-4b1a-b89d-1c86db34f703",
   "metadata": {},
   "outputs": [],
   "source": [
    "rainfall_df.to_parquet('../data/processed/brooklyn/brooklyn-rainfall-2021.parq')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4adb364-1ace-4f65-b9cb-2b74b19556aa",
   "metadata": {},
   "source": [
    "So, plain and simple, short and sweet.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21704f83-2b0c-4f01-9da8-482ddbf3d6a1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}