From ba2801a4f9b7fcc9305ef58b74fa30aee8fdf8c7 Mon Sep 17 00:00:00 2001 From: Lakshay-sethi <58126894+Lakshay-sethi@users.noreply.github.com> Date: Tue, 20 Apr 2021 10:47:15 -0700 Subject: [PATCH 1/2] added Covid tracking example --- examples/DataConnector_CovidTracking.ipynb | 149 +++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 examples/DataConnector_CovidTracking.ipynb diff --git a/examples/DataConnector_CovidTracking.ipynb b/examples/DataConnector_CovidTracking.ipynb new file mode 100644 index 000000000..d7770eb15 --- /dev/null +++ b/examples/DataConnector_CovidTracking.ipynb @@ -0,0 +1,149 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Connector for Covid Tracking\n", + "\n", + "In this example, we will be going over how to use Connector with Covid Tarcking and understand how to use the Dynamic Url functionality" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "Connector is a component in the DataPrep library that aims to simplify data access by providing a standard API set. The goal is to help users skip the complex API configuration. In this tutorial, we demonstrate how to use the connector component with Twitter.\n", + "\n", + "If you haven't installed DataPrep, run command `!pip install dataprep` or execute the following cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Run me if you'd like to install\n", + "!pip install dataprep" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Download and store the configuration files in DataPrep. \n", + "\n", + "The configuration files are used to configure the parameters and initial setup for the API. The available configuration files can be manually downloaded here: [Configuration Files](https://github.com/sfu-db/DataConnectorConfigs) or automatically downloaded at usage. \n", + "\n", + "Store the configuration file in the dataprep folder. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Initialize connector\n", + "\n", + "This establishes a connection with Covid Tracking Api and returns an object. Once you run the code you can use the built in functions available from connector." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from dataprep.connector import connect, info\n", + "\n", + "dc = connect('covidTracking', _concurrency = 10)\n", + "\n", + "dc" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Functionalities\n", + "\n", + "Connector has several functions you can perform to gain insight on the data downloaded from Covid Tarcking API." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Connector.query\n", + "The query method downloads the website data and displays it in a Dataframe. The parameters must meet the requirements as indicated in connector.info for the operation to run. \n", + "\n", + "When the data is received from the server, it will either be in a JSON or XML format. The connector reformats the data in pandas Dataframe for the convenience of downstream operations.\n", + "\n" + ] + }, + { + "source": [ + "### Dynamic Url Creation\n", + "The Url genration method allows the user to create a url in which url can be chnaged to allow on the spot modification. The parameters passed from the query must meet the requirements as indicated in url for the operation to run.\n", + "\n", + "For Example:\n", + "\n", + "http://api.covidtracking.com/v2/states/{state}/daily.json to http://api.covidtracking.com/v2/states/ny/daily.json \n", + "\n", + "where the state parameter is replaced by ny (Abbr. New York)\n", + "\n", + "Now, let's try to get Historical COVID-19 data for a state in US from Covid Data API." + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Searching for Historical COVID-19 data in Michigan" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Getting covid data for Michigan\n", + "df = await dc.query(\"state_historical\",state = \"mi\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# That's all for now. \n", + "If you are interested in writing your own configuration file or modify an existing one, refer to the [Example folder](https://github.com/sfu-db/DataConnectorConfigs>). Change the link" + ] + } + ], + "metadata": { + "kernelspec": { + "name": "python376jvsc74a57bd0c958774743908812a06d5249a6a57af991212914420a506af38b26bbd3d5952a", + "display_name": "Python 3.7.6 64-bit" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6-final" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file From 7b201eb09e6968f1ff56cb6385f9cbe9e523d581 Mon Sep 17 00:00:00 2001 From: Lakshay-sethi <58126894+Lakshay-sethi@users.noreply.github.com> Date: Thu, 22 Apr 2021 22:27:04 -0700 Subject: [PATCH 2/2] changes to the documentation --- docs/source/user_guide/connector/query.ipynb | 12 ++++++++-- examples/DataConnector_CovidTracking.ipynb | 23 +++++++++++++++----- 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/docs/source/user_guide/connector/query.ipynb b/docs/source/user_guide/connector/query.ipynb index a3b17da57..4df77cbf5 100644 --- a/docs/source/user_guide/connector/query.ipynb +++ b/docs/source/user_guide/connector/query.ipynb @@ -81,7 +81,15 @@ "```\n", "df = await dc.query(\"tweets\", _q=\"COVID-19\", _count=50)\n", "```\n", - "This query searches 50 results for tweets related to COVID-19 from Twitter search API." + "This query searches 50 results for tweets related to COVID-19 from Twitter search API.\n", + "\n", + "\n", + "* **Covid Tracking**\n", + "\n", + "```\n", + "df = await dc.query(\"state_historical\", state = \"mi\")\n", + "```\n", + "This query searches for Historical COVID-19 data in Michigan" ] } ], @@ -106,4 +114,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/examples/DataConnector_CovidTracking.ipynb b/examples/DataConnector_CovidTracking.ipynb index d7770eb15..f39e5df8b 100644 --- a/examples/DataConnector_CovidTracking.ipynb +++ b/examples/DataConnector_CovidTracking.ipynb @@ -86,19 +86,32 @@ { "source": [ "### Dynamic Url Creation\n", - "The Url genration method allows the user to create a url in which url can be chnaged to allow on the spot modification. The parameters passed from the query must meet the requirements as indicated in url for the operation to run.\n", + "The Url genration method allows the user to create a url in which url can be changed to allow on the spot modification. The parameters passed from the query must meet the requirements as indicated in url for the operation to run.\n", "\n", "For Example:\n", "\n", - "http://api.covidtracking.com/v2/states/{state}/daily.json to http://api.covidtracking.com/v2/states/ny/daily.json \n", + "We have the following url, here you can see that, we have the ability to modify the url with any state we want.\n", + "http://api.covidtracking.com/v2/states/{state}/current.json \n", "\n", - "where the state parameter is replaced by ny (Abbr. New York)\n", + "Lets suppose we want to retrieve data for New York. We would just inter-exchange the {state} variable with ny (Abbr. for New York)\n", + "http://api.covidtracking.com/v2/states/ny/current.json\n", "\n", - "Now, let's try to get Historical COVID-19 data for a state in US from Covid Data API." + "Now seeing this you may be asking where do we pass the data we want to replace. This can be achived from by pacing the data as a parameter in the query function.\n", + "\n", + "Example:\n", + "\n", + "await dc.query(\"states\",state = \"ny\")\n", + "\n", + "Now, let's try to get Current COVID-19 data for a state in US from Covid Data API." ], "cell_type": "markdown", "metadata": {} }, + { + "source": [], + "cell_type": "markdown", + "metadata": {} + }, { "cell_type": "markdown", "metadata": {}, @@ -122,7 +135,7 @@ "metadata": {}, "source": [ "# That's all for now. \n", - "If you are interested in writing your own configuration file or modify an existing one, refer to the [Example folder](https://github.com/sfu-db/DataConnectorConfigs>). Change the link" + "If you are interested in writing your own configuration file or modify an existing one, refer to the [Example folder]()." ] } ],