Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc (Connector) : Addition of a ipynb script for Dynamic Url #586

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/source/user_guide/connector/query.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,15 @@
"```\n",
"df = await dc.query(\"tweets\", _q=\"COVID-19\", _count=50)\n",
"```\n",
"This query searches 50 results for tweets related to COVID-19 from Twitter search API."
"This query searches 50 results for tweets related to COVID-19 from Twitter search API.\n",
"\n",
"\n",
"* **Covid Tracking**\n",
"\n",
"```\n",
"df = await dc.query(\"state_historical\", state = \"mi\")\n",
"```\n",
"This query searches for Historical COVID-19 data in Michigan"
]
}
],
Expand All @@ -106,4 +114,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}
162 changes: 162 additions & 0 deletions examples/DataConnector_CovidTracking.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Connector for Covid Tracking\n",
"\n",
"In this example, we will be going over how to use Connector with Covid Tarcking and understand how to use the Dynamic Url functionality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Connector is a component in the DataPrep library that aims to simplify data access by providing a standard API set. The goal is to help users skip the complex API configuration. In this tutorial, we demonstrate how to use the connector component with Twitter.\n",
"\n",
"If you haven't installed DataPrep, run command `!pip install dataprep` or execute the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Run me if you'd like to install\n",
"!pip install dataprep"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Download and store the configuration files in DataPrep. \n",
"\n",
"The configuration files are used to configure the parameters and initial setup for the API. The available configuration files can be manually downloaded here: [Configuration Files](https://github.com/sfu-db/DataConnectorConfigs) or automatically downloaded at usage. \n",
"\n",
"Store the configuration file in the dataprep folder. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Initialize connector\n",
"\n",
"This establishes a connection with Covid Tracking Api and returns an object. Once you run the code you can use the built in functions available from connector."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from dataprep.connector import connect, info\n",
"\n",
"dc = connect('covidTracking', _concurrency = 10)\n",
"\n",
"dc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Functionalities\n",
"\n",
"Connector has several functions you can perform to gain insight on the data downloaded from Covid Tarcking API."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Connector.query\n",
"The query method downloads the website data and displays it in a Dataframe. The parameters must meet the requirements as indicated in connector.info for the operation to run. \n",
"\n",
"When the data is received from the server, it will either be in a JSON or XML format. The connector reformats the data in pandas Dataframe for the convenience of downstream operations.\n",
"\n"
]
},
{
"source": [
"### Dynamic Url Creation\n",
"The Url genration method allows the user to create a url in which url can be changed to allow on the spot modification. The parameters passed from the query must meet the requirements as indicated in url for the operation to run.\n",
"\n",
"For Example:\n",
"\n",
"We have the following url, here you can see that, we have the ability to modify the url with any state we want.\n",
"http://api.covidtracking.com/v2/states/{state}/current.json \n",
"\n",
"Lets suppose we want to retrieve data for New York. We would just inter-exchange the {state} variable with ny (Abbr. for New York)\n",
"http://api.covidtracking.com/v2/states/ny/current.json\n",
"\n",
"Now seeing this you may be asking where do we pass the data we want to replace. This can be achived from by pacing the data as a parameter in the query function.\n",
"\n",
"Example:\n",
"\n",
"await dc.query(\"states\",state = \"ny\")\n",
"\n",
"Now, let's try to get Current COVID-19 data for a state in US from Covid Data API."
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching for Historical COVID-19 data in Michigan"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Getting covid data for Michigan\n",
"df = await dc.query(\"state_historical\",state = \"mi\")\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# That's all for now. \n",
"If you are interested in writing your own configuration file or modify an existing one, refer to the [Example folder](<https://github.com/sfu-db/APIConnectors/tree/examples/examples>)."
]
}
],
"metadata": {
"kernelspec": {
"name": "python376jvsc74a57bd0c958774743908812a06d5249a6a57af991212914420a506af38b26bbd3d5952a",
"display_name": "Python 3.7.6 64-bit"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6-final"
}
},
"nbformat": 4,
"nbformat_minor": 4
}