diff --git a/api/index.md b/api/index.md index 49a70a9a3..cde9e6cb7 100644 --- a/api/index.md +++ b/api/index.md @@ -1,25 +1,24 @@ --- layout: default title: API -nav_order: 20 +nav_order: 0 has_children: true --- -# Overview +# API overview [Data Commons](https://datacommons.org){: target="_blank"} aggregates data from many different [data sources](https://datacommons.org/datasets){: target="_blank"} into a single database. Data Commons is based on the data model used by [schema.org](https://schema.org){: target="_blank"}; for more information, see [Key concepts](/data_model.html). -The Data Commons APIs allow developers to programmatically access the data in Data Commons. -Data Commons provides several different ways to access its resources: +The Data Commons APIs allow developers to programmatically access the data in Data Commons, using the following technologies: * A [REST API](/api/rest/v2) that can be used on the command line as well as in any language with an HTTP library. * [Python](/api/python) and [Pandas](/api/pandas) wrappers. -> **Note:** The Python and Pandas APIs wrap the [v1](/api/rest/v1) version of the REST APIs and have not yet been updated to v2. +> **Note:** The Python and Pandas APIs wrap the [V1](/api/rest/v1) version of the REST APIs and have not yet been updated to v2. The endpoints can be roughly grouped into four categories: @@ -33,4 +32,35 @@ The endpoints can be roughly grouped into four categories: graph query language [SPARQL](https://www.w3.org/TR/rdf-sparql-query/){: target="_blank"}. This is useful for complex node connections which would require multiple API calls; for example, "hate crimes motivated by disability status in Californian cities". - **Utilities**: These are Python notebook-specific APIs for helping with - Pandas DataFrames, etc. \ No newline at end of file + Pandas DataFrames, etc. + +In addition, Data Commons provides additional tools for accessing its data that call the REST APIs under the hood: + +- [Google Sheets](sheets/index.md): provides several custom functions that populate spreadsheets with data from the Data Commons knowledge graph +- [Web Components](web_components/index.md): provides JavaScript APIs and HTML templates that allow you to embed Data Commons data and visualizations into web pages + + +## API keys {: #get-key} + +A key is required by some APIs to authenticate and authorize requests. +- All REST [V2](rest/v2/index.md) and [V1](rest/v1/index.md) APIs. These requests are served by endpoints at `api.datacommons.org`. +- All requests coming from a custom Data Commons instance. These are also served by `api.datacommons.org`. +- Data Commons NL API requests (used by the [DataGemma](https://ai.google.devgit/gemma/docs/datagemma){: target="_blank"} tool). These are served by endpoints at `nl.datacommons.org`. + +A key is currently not required for the following, although this may change in the future: +- Python and Pandas client libraries other than NL APIs +- V0 REST APIs +- Google Sheets +- Web Components + +### Obtain an API key + +Data Commons API keys are managed by a self-service portal. To obtain an API key, go to [https://apikeys.datacommons.org](https://apikeys.datacommons.org){: target="_blank"} and request a key for the hostname(s) listed above. Enable each of the APIs you want; you can share a single key for all of them. + +To use the key in requests, see the relevant documentation: +- For REST V2 APIs, see the section on [Authentication](/api/rest/v2/index.html#authentication). +- For REST V1 APIs, see the section on [Authentication](/api/rest/v2/getting_started.html#authentication). +- For NL APIs in DataGemma, see the Colab notebooks in [https://github.com/datacommonsorg/llm-tools/tree/main/notebooks](https://github.com/datacommonsorg/llm-tools/tree/main/notebooks){: target="_blank"} + + + diff --git a/api/pandas/index.md b/api/pandas/index.md index 78204c292..c6434b7e7 100644 --- a/api/pandas/index.md +++ b/api/pandas/index.md @@ -1,7 +1,7 @@ --- layout: default title: Pandas -nav_order: 30 +nav_order: 50 parent: API has_children: true --- @@ -15,7 +15,7 @@ the Pandas API, and supplemental functions help with directly creating objects using data from the Data Commons knowledge graph for common use cases. -> **Note:** The Pandas API only supports [v1](/api/rest/v1/index.html) of the REST APIs. +> **Note:** The Pandas API only supports [V1](/api/rest/v1/index.html) of the REST APIs. Before proceeding, make sure you have followed the setup instructions below. @@ -27,7 +27,7 @@ Before proceeding, make sure you have followed the setup instructions below. ```bash $ pip install datacommons_pandas ``` -You are ready to go! You can view our [tutorials](tutorials.md) on how to use the +You are ready to go! You can view our [tutorials](/api/python/tutorials.html) on how to use the API to perform certain tasks using [Google Colab](https://colab.sandbox.google.com/){: target="_blank"}, or refer to pages in the navigation bar for detailed information about all the methods available. ## Run Python interactively diff --git a/api/python/index.md b/api/python/index.md index 24312359d..3513d93c2 100644 --- a/api/python/index.md +++ b/api/python/index.md @@ -1,7 +1,7 @@ --- layout: default title: Python -nav_order: 20 +nav_order: 40 parent: API has_children: true --- @@ -13,7 +13,7 @@ programmatically access nodes in the Data Commons knowledge graph. This package allows users to explore the structure of the graph, integrate statistics from the graph into data analysis workflows and much more. -> **Note:** The Python API only supports [v1](/api/rest/v1) of the REST APIs. +> **Note:** The Python API only supports [V1](/api/rest/v1) of the REST APIs. Before proceeding, make sure you have followed the setup instructions below. diff --git a/api/python/query.md b/api/python/query.md index 8a6889293..e39e2c7ea 100644 --- a/api/python/query.md +++ b/api/python/query.md @@ -12,7 +12,7 @@ Returns the results of running a graph query on the Data Commons knowledge graph using [SPARQL](https://www.w3.org/TR/rdf-sparql-query/){: target="_blank"}. Note that Data Commons is only able to support a limited subsection of SPARQL functionality at this time: specifically only the keywords `ORDER BY`, `DISTINCT`, and `LIMIT`. -Note: The Python SPARQL library currently only supports the [v1](/api/v1/query.html) version of the API. +Note: The Python SPARQL library currently only supports the [V1](/api/v1/query.html) version of the API. ## General information about the query() method diff --git a/api/python/tutorials.md b/api/python/tutorials.md index 82429bb63..779a64f10 100644 --- a/api/python/tutorials.md +++ b/api/python/tutorials.md @@ -17,16 +17,16 @@ Python: - [Getting Started: Analyzing Census Data](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_census_data.ipynb){:target="_blank"} -- [Case Study: COVID-19 Feature Exploration Analysis (by Google Health)](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/COVID_19_Feature_Exploration_Analysis_with_Data_Commons.ipynb){:target="_blank"} +- [COVID-19 Feature Exploration Analysis (by Google Health)](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/COVID_19_Feature_Exploration_Analysis_with_Data_Commons.ipynb){:target="_blank"} -- [Case Study: Analyzing Income Distribution](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_income_distribution.ipynb){:target="_blank"} +- [Analyzing Income Distribution](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_income_distribution.ipynb){:target="_blank"} -- [Case Study: Prevalence of Obesity in 500 US Cities](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_obesity_prevalence.ipynb){:target="_blank"} +- [Prevalence of Obesity in 500 US Cities](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_obesity_prevalence.ipynb){:target="_blank"} -- [Case Study: Analyzing Genomic Data with Biomedical Data Commons](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_genomic_data.ipynb){:target="_blank"} +- [Analyzing Genomic Data with Biomedical Data Commons](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/analyzing_genomic_data.ipynb){:target="_blank"} -- [Case Study: Drug Discovery with Biomedical Data Commons](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Drug_Discovery_With_Data_Commons.ipynb){:target="_blank"} +- [Drug Discovery with Biomedical Data Commons](https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Drug_Discovery_With_Data_Commons.ipynb){:target="_blank"} -- [Case Study: Analyzing Superfund Sites with Data Commons](https://colab.sandbox.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Analyzing_SuperfundSites_with_Data_Commons.ipynb){:target="_blank"} +- [Analyzing Superfund Sites with Data Commons](https://colab.sandbox.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Analyzing_SuperfundSites_with_Data_Commons.ipynb){:target="_blank"} -- [Case Study: Estimating CMIP6 Temperature Distributions](https://colab.sandbox.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Estimating_(Temperature)_Distributions_With_DataCommons_.ipynb){:target="_blank"} \ No newline at end of file +- [Estimating CMIP6 Temperature Distributions](https://colab.sandbox.google.com/github/datacommonsorg/api-python/blob/master/notebooks/Estimating_(Temperature)_Distributions_With_DataCommons_.ipynb){:target="_blank"} \ No newline at end of file diff --git a/api/rest/v1/getting_started.md b/api/rest/v1/getting_started.md index 0a157dc23..9af4bce47 100644 --- a/api/rest/v1/getting_started.md +++ b/api/rest/v1/getting_started.md @@ -157,8 +157,7 @@ We've provided a trial API key for general public use. This key will let you try `AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI` -The trial key is capped with a limited quota for requests. If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please request one by -[filling out this form](https://docs.google.com/forms/d/e/1FAIpQLSeVCR95YOZ56ABsPwdH1tPAjjIeVDtisLF-8oDYlOxYmNZ7LQ/viewform) and selecting "API access" to request an official key without any quota limits. We'll be happy to hear from you! +The trial key is capped with a limited quota for requests. If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please go to the portal at https://apikeys.datacommons.org and request a key for `api.datacommons.org`. ### Pagination {: #pagination} diff --git a/api/rest/v2/getting_started.md b/api/rest/v2/getting_started.md index a3816a3e4..51d19e5df 100644 --- a/api/rest/v2/getting_started.md +++ b/api/rest/v2/getting_started.md @@ -1,225 +1,5 @@ --- -layout: default -title: Getting started -nav_order: 0 -parent: REST (v2) -grand_parent: API -published: true +layout: redirect +redirect: /api/rest/v2/index.html +nav_exclude: true --- - -{:.no_toc} -# Getting started - -* TOC -{:toc} - -Following HTTP, a REST API call consists of a _request_ that you provide, and a _response_ from the Data Commons servers with the data you requested, in [JSON](https://json.org){: target="_blank"} format. The following sections detail how to assemble a request. - -## Service endpoints - -You make requests through [API endpoints](https://en.wikipedia.org/wiki/Web_API#Endpoints){: target="_blank"}. You access each endpoint using its unique URL, which is a combination of a base URL and the endpoint's [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier){: target="_blank"}. - -The base URL for all REST endpoints is: - -
-https://api.datacommons.org/VERSION
-
- -The current version is `v2`. - -To access a particular endpoint, append the URI to the base URL (e.g. `https://api.datacommons.org/v2/node` ). -The URIs for the V2 API are below: - -| API | URI path | Description | -| --- | --- | ----------- | -| Node | [/node](/api/rest/v2/node) | Fetches information about edges and neighboring nodes | -| Observation | [/observation](/api/rest/v2/observation) | Fetches statistical observations | -| Resolve entities | [/resolve](/api/rest/v2/resolve) | Returns a Data Commons ID ([`DCID`](/glossary.html#dcid)) for entities in the graph | -| SPARQL | [/v2/sparql](/sparql) | Returns matches to a [SPARQL](https://www.w3.org/TR/rdf-sparql-query/){: target="_blank"} graph query | - -### Endpoints for custom instances - -If you are running your own Data Commons, the URL/URI endpoints are slightly different: - -
-CUSTOM_URL/core/api/v2
-
- -## Query parameters {#query-param} - -Endpoints take a set of parameters which allow you to specify the entities, variables, timescales, etc. you are interested in. The V2 APIs only use query parameters. - -Query parameters are chained at the end of a URL behind a `?` symbol. Separate multiple parameter entries with an `&` symbol. For example, this would look like: - -
-https://api.datacommons.org/v2/node?key=API_KEY&nodes=DCID1&nodes=DCID2&property=<-*
-
- -Still confused? Each endpoint's documentation page has examples at the bottom tailored to the endpoint you're trying to use. - -## POST requests - -All V2 endpoints allow for POST requests. For POST requests, feed all parameters in JSON format. For example, in cURL, this would look like: - -
-curl -X POST \
--H "X-API-Key: API_KEY" \
---url https://api.datacommons.org/v2/node \
---data '{
-  "nodes": [
-    "geoId/06085",
-    "geoId/06086"
-  ],
-  "property": "->[name, latitude, longitude]"
-}'
-
- - -{: #authentication} -## Authentication - -API keys are required in any REST API request. To obtain an API key, please see [Get API key](/api//rest/v2/index.html#get-key). - -> **Note:** If you are sending API requests to a custom Data Commons instance, do _not_ include any API key in the requests. - -To include an API key, add your API key to the URL as a query parameter by appending ?key=API_KEY. - -For GET requests, this looks like: - -
-https://api.datacommons.org/v2/ENDPOINT?key=API_KEY
-
- -If the key is not the first query parameter, use &key=API_KEY instead. This looks like: - -
-https://api.datacommons.org/v2/ENDPOINT?QUERY=VALUE&key=API_KEY
-
- -For POST requests, pass the key as a header. For example, in cURL, this looks like: - -
-curl -X POST \
---url https://api.datacommons.org/v2/node \
---header 'X-API-Key: API_KEY' \
---data '{
-  "nodes": [
-    "ENTITY_DCID_1",
-    "ENTITY_DCID_2",
-    ...
-  ],
-  "property: "RELATION_EXPRESSION"
-}'
-
- -{: #pagination} -## Pagination - -When the response to a request is too long, the returned payload is -_paginated_. Only a subset of the response is returned, along with a long string -of characters called a _token_. To get the next set of entries, repeat the -request with `nextToken` as an query parameter, with the token as its value. - -For example, the request: - -```bash -curl --request GET \ - 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*' -``` - -will return something like: - -```json -{ - "data": { - "geoId/06": { - "arcs": < ... output truncated for brevity ...> - }, - }, - "nextToken": "SoME_veRy_L0ng_S+rIng" -} -``` - -To get the next set of entries, repeat the previous command and append the `nextToken`: - -```bash -curl --request GET \ - 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*&nextToken=SoME_veRy_L0ng_S+rIng' -``` - -Similarly for POST requests, this would look like: - -```bash -curl -X POST \ --H "X-API-Key: AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI" \ ---url https://api.datacommons.org/v2/node \ ---data '{ - "nodes": "geoId/06", - "property": "<-*", - "nextToken": "SoME_veRy_L0ng_S+rIng" -}' -``` - -{: #relation-expressions} -## Relation expressions - -Data Commons represents real world entities and data as nodes. These -nodes are connected by directed edges, or arcs, to form a knowledge graph. The -label of the arc is the name of the [property](/glossary.html#property). - -Relation expressions include arrow annotation and other symbols in the syntax to -represent neighboring nodes, and to support chaining and filtering. -These new expressions allow all of the functionality of the V1 API to be -expressed with fewer API endpoints in V2. All V2 API calls require relation -expressions in the `property` or `expression` parameter. - -The following table describes symbols in the V2 API relation expressions: - -| ------ | ---------- | -| `->` | An outgoing arc | -| `<-` | An incoming arc | -| {PROPERTY:VALUE} | Filtering; identifies the property and associated value | -| `[]` | Multiple properties, separated by commas | -| `*` | All properties linked to this node | -| `+` | One or more expressions chained together for indirect relationships, like `containedInPlace+{typeOf:City}` | - -### Incoming and outgoing arcs - -Arcs in the Data Commons Graph have directions. In the example below, for the node [Argentina](https://datacommons.org/browser/country/ARG){: target="_blank"}, the property `containedInPlace` exists in both in and out directions, illustrated in the following figure: - -![](/assets/images/rest/property_value_direction_example.png) - -Note the directionality of the property `containedInPlace`: incoming arc represents "Argentina contains Buenos Aires", while the outgoing arc represents "Argentina is in South America".* - -Nodes for outgoing arcs are represented by `->`, while nodes for incoming arcs -arcs are represented by `<-`. To illustrate using the above example: - -- Regions that include Argentina (DCID: `country/ARG`): `country/ARG->containedInPlace` -- All cities directly contained in Argentina (DCID: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` - -### Filters - -You can use filters to reduce results to only match nodes with a specified property and value. Use {} to specify property:value pairs to define the filter. Using the same example, `country/ARG<-containedInPlace+{typeOf:City}` only returns nodes with the `typeOf:City`, filtering out `typeOf:AdministrativeArea1` and so on. - -### Specify multiple properties - -You can combine multiple properties together within `[]`. For example, to request a few outgoing arcs for a node, use -`->[name, latitude, longitude]`. See more in this [Node API example](/api/rest/v2/node.html#multiple-properties)). - -### Wildcard - -To retrieve all properties linked to a node, use the `*` wildcard, e.g. `<-*`. -See more in this [Node API example](/api/rest/v2/node.html#wildcard). - -### Chain properties - -Use `+` to express a chain expression. A chain expression represents requests for information about nodes -which are connected by the same property, but are a few hops away. This is supported only for the `containedInPlace` property. - -To illustrate again using the Argentina example: -- All cities directly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` -- All cities indirectly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace+{typeOf:City}` - -## Escape codes for reserved characters in GET requests - -HTTP GET requests do not allow some of the characters used by Data Commons DCIDs and relation expressions. When sending GET requests, you may need use the [corresponding percent codes](https://en.wikipedia.org/wiki/Percent-encoding){: target="_blank"} for reserved characters. \ No newline at end of file diff --git a/api/rest/v2/index.md b/api/rest/v2/index.md index 79d2320ff..35dc5b622 100644 --- a/api/rest/v2/index.md +++ b/api/rest/v2/index.md @@ -1,14 +1,14 @@ --- layout: default -title: REST (v2) -nav_order: 0 +title: REST (V2) +nav_order: 10 parent: API has_children: true published: true --- {:.no_toc} -# Data Commons REST API +# Data Commons REST API V2 * TOC {:toc} @@ -22,16 +22,75 @@ knowledge graph, using [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_P graph, integrate statistics from the graph into data analysis applications and much more. -You can use the REST API with any tool or language that supports HTTP. You can make queries on the command line (e.g. using [cURL](https://curl.se/){: target="_blank"}), by scripting HTTP requests in another language like Javascript, or even by entering an endpoint into your web browser! +Following HTTP, a REST API call consists of a _request_ that you provide, and a _response_ from the Data Commons servers with the data you requested, in [JSON](https://json.org){: target="_blank"} format. You can use the REST API with any tool or language that supports HTTP. You can make queries on the command line (e.g. using [cURL](https://curl.se/){: target="_blank"}), by scripting HTTP requests in another language like Javascript, or even by entering an endpoint into your web browser! ## What's new in V2 -The V2 API collapses functionality from [V1 API](/api/rest/v1) into a smaller number of endpoints, by introducing a syntax for _relation expressions_, described [here](/api/rest/v2/getting_started.html#relation-expressions). Each API endpoint can also handle both single and bulk requests. +The V2 API collapses functionality from the [V1 API](/api/rest/v1) into a smaller number of endpoints, by introducing a syntax for _relation expressions_, described below. Each API endpoint can also handle both single and bulk requests. -{: #get-key} -## Get API keys +## Service endpoints -All access to base Data Commons using the REST APIs must be authenticated and authorized with an API key. (To send API requests to custom Data Commons instances, no API key is required.) +You make requests through [API endpoints](https://en.wikipedia.org/wiki/Web_API#Endpoints){: target="_blank"}. You access each endpoint using its unique URL, which is a combination of a base URL and the endpoint's [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier){: target="_blank"}. + +The base URL for all REST endpoints is: + +
+https://api.datacommons.org/VERSION
+
+ +The current version is `v2`. + +To access a particular endpoint, append the URI to the base URL (e.g. `https://api.datacommons.org/v2/node` ). +The URIs for the V2 API are below: + +| API | URI path | Description | +| --- | --- | ----------- | +| Node | [/node](/api/rest/v2/node) | Fetches information about edges and neighboring nodes | +| Observation | [/observation](/api/rest/v2/observation) | Fetches statistical observations | +| Resolve entities | [/resolve](/api/rest/v2/resolve) | Returns a Data Commons ID ([`DCID`](/glossary.html#dcid)) for entities in the graph | +| SPARQL | [/v2/sparql](/sparql) | Returns matches to a [SPARQL](https://www.w3.org/TR/rdf-sparql-query/){: target="_blank"} graph query | + +### Endpoints for custom instances + +If you are running your own Data Commons, the URL/URI endpoints are slightly different: + +
+CUSTOM_URL/core/api/v2
+
+ +## Query parameters {#query-param} + +Endpoints take a set of parameters which allow you to specify the entities, variables, timescales, etc. you are interested in. The V2 APIs only use query parameters. + +Query parameters are chained at the end of a URL behind a `?` symbol. Separate multiple parameter entries with an `&` symbol. For example, this would look like: + +
+https://api.datacommons.org/v2/node?key=API_KEY&nodes=DCID1&nodes=DCID2&property=<-*
+
+ +Still confused? Each endpoint's documentation page has examples at the bottom tailored to the endpoint you're trying to use. + +## POST requests + +All V2 endpoints allow for POST requests. For POST requests, feed all parameters in JSON format. For example, in cURL, this would look like: + +
+curl -X POST \
+-H "X-API-Key: API_KEY" \
+--url https://api.datacommons.org/v2/node \
+--data '{
+  "nodes": [
+    "geoId/06085",
+    "geoId/06086"
+  ],
+  "property": "->[name, latitude, longitude]"
+}'
+
+ +{: #authentication} +## Authentication + +All access to base Data Commons using the REST APIs must be authenticated and authorized with an API key. We provide a trial API key for general public use. This key will let you try the API and make single requests. @@ -40,14 +99,155 @@ We provide a trial API key for general public use. This key will let you try the `AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI` -_The trial key is capped with a limited quota for requests._ If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please request an official key without any quota limits by -[filling out this form](https://docs.google.com/forms/d/e/1FAIpQLSeVCR95YOZ56ABsPwdH1tPAjjIeVDtisLF-8oDYlOxYmNZ7LQ/viewform?usp=dialog){: target="_blank"}. Typical turnaround times are 24-48 hours. +_The trial key is capped with a limited quota for requests._ If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please request an official key without any quota limits; please see [Obtain an API key](/api/index.html#get-key) for information. + +> **Note:** If you are sending API requests to a custom Data Commons instance, do _not_ include any API key in the requests. + +To include an API key, add your API key to the URL as a query parameter by appending ?key=API_KEY. + +For GET requests, this looks like: + +
+https://api.datacommons.org/v2/ENDPOINT?key=API_KEY
+
-To use the key in requests, see the [Authentication](/api/rest/v2/getting_started.html#authentication) section. +If the key is not the first query parameter, use &key=API_KEY instead. This looks like: + +
+https://api.datacommons.org/v2/ENDPOINT?QUERY=VALUE&key=API_KEY
+
+ +For POST requests, pass the key as a header. For example, in cURL, this looks like: + +
+curl -X POST \
+--url https://api.datacommons.org/v2/node \
+--header 'X-API-Key: API_KEY' \
+--data '{
+  "nodes": [
+    "ENTITY_DCID_1",
+    "ENTITY_DCID_2",
+    ...
+  ],
+  "property: "RELATION_EXPRESSION"
+}'
+
## Find available entities, variables, and their DCIDs Many requests require the [DCID](/glossary.html#dcid) of the entity or variable you wish to query. For tips on how to find relevant DCIDs, entities and variables, please see the [Key concepts](/data_model.html) document, specifically the following sections: - [Find a DCID for an entity or variable](/data_model.html#find-dcid) -- [Find places available for a statistical variable](/data_model.html#find-places) \ No newline at end of file +- [Find places available for a statistical variable](/data_model.html#find-places) + +{: #relation-expressions} +## Relation expressions + +Data Commons represents real world entities and data as nodes. These +nodes are connected by directed edges, or arcs, to form a knowledge graph. The +label of the arc is the name of the [property](/glossary.html#property). + +Relation expressions include arrow annotation and other symbols in the syntax to +represent neighboring nodes, and to support chaining and filtering. +These new expressions allow all of the functionality of the V1 API to be +expressed with fewer API endpoints in V2. All V2 API calls require relation +expressions in the `property` or `expression` parameter. + +The following table describes symbols in the V2 API relation expressions: + +| ------ | ---------- | +| `->` | An outgoing arc | +| `<-` | An incoming arc | +| {PROPERTY:VALUE} | Filtering; identifies the property and associated value | +| `[]` | Multiple properties, separated by commas | +| `*` | All properties linked to this node | +| `+` | One or more expressions chained together for indirect relationships, like `containedInPlace+{typeOf:City}` | + +### Incoming and outgoing arcs + +Arcs in the Data Commons Graph have directions. In the example below, for the node [Argentina](https://datacommons.org/browser/country/ARG){: target="_blank"}, the property `containedInPlace` exists in both in and out directions, illustrated in the following figure: + +![](/assets/images/rest/property_value_direction_example.png) + +Note the directionality of the property `containedInPlace`: incoming arc represents "Argentina contains Buenos Aires", while the outgoing arc represents "Argentina is in South America".* + +Nodes for outgoing arcs are represented by `->`, while nodes for incoming arcs +arcs are represented by `<-`. To illustrate using the above example: + +- Regions that include Argentina (DCID: `country/ARG`): `country/ARG->containedInPlace` +- All cities directly contained in Argentina (DCID: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` + +### Filters + +You can use filters to reduce results to only match nodes with a specified property and value. Use {} to specify property:value pairs to define the filter. Using the same example, `country/ARG<-containedInPlace+{typeOf:City}` only returns nodes with the `typeOf:City`, filtering out `typeOf:AdministrativeArea1` and so on. + +### Specify multiple properties + +You can combine multiple properties together within `[]`. For example, to request a few outgoing arcs for a node, use +`->[name, latitude, longitude]`. See more in this [Node API example](/api/rest/v2/node.html#multiple-properties)). + +### Wildcard + +To retrieve all properties linked to a node, use the `*` wildcard, e.g. `<-*`. +See more in this [Node API example](/api/rest/v2/node.html#wildcard). + +### Chain properties + +Use `+` to express a chain expression. A chain expression represents requests for information about nodes +which are connected by the same property, but are a few hops away. This is supported only for the `containedInPlace` property. + +To illustrate again using the Argentina example: +- All cities directly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` +- All cities indirectly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace+{typeOf:City}` + +## Escape codes for reserved characters in GET requests + +HTTP GET requests do not allow some of the characters used by Data Commons DCIDs and relation expressions. When sending GET requests, you may need use the [corresponding percent codes](https://en.wikipedia.org/wiki/Percent-encoding){: target="_blank"} for reserved characters. + +{: #pagination} +## Pagination + +When the response to a request is too long, the returned payload is +_paginated_. Only a subset of the response is returned, along with a long string +of characters called a _token_. To get the next set of entries, repeat the +request with `nextToken` as an query parameter, with the token as its value. + +For example, the request: + +```bash +curl --request GET \ + 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*' +``` + +will return something like: + +```json +{ + "data": { + "geoId/06": { + "arcs": < ... output truncated for brevity ...> + }, + }, + "nextToken": "SoME_veRy_L0ng_S+rIng" +} +``` + +To get the next set of entries, repeat the previous command and append the `nextToken`: + +```bash +curl --request GET \ + 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*&nextToken=SoME_veRy_L0ng_S+rIng' +``` + +Similarly for POST requests, this would look like: + +```bash +curl -X POST \ +-H "X-API-Key: AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI" \ +--url https://api.datacommons.org/v2/node \ +--data '{ + "nodes": "geoId/06", + "property": "<-*", + "nextToken": "SoME_veRy_L0ng_S+rIng" +}' +``` \ No newline at end of file diff --git a/api/rest/v2/node.md b/api/rest/v2/node.md index dbec9e34c..a336bc093 100644 --- a/api/rest/v2/node.md +++ b/api/rest/v2/node.md @@ -2,7 +2,7 @@ layout: default title: Get node properties nav_order: 3 -parent: REST (v2) +parent: REST (V2) grand_parent: API published: true --- @@ -62,7 +62,7 @@ JSON data: | Name | Type | Description | | ----------------------------------------------------- | ------ | -----------------------| -| key
Required | string | Your API key. See the section on [authentication](/api/rest/v2/getting_started.html#authentication) details. | +| key
Required | string | Your API key. See the section on [authentication](/api/rest/v2/index.html#authentication) for details. | | nodes
Required | list of strings | List of the [DCIDs](/glossary.html#dcid) of the nodes to query. | | property
Required | string | Property to query, represented with symbols including arrow notation. For more details, see [relation expressions](/api/rest/v2/#relation-expressions). By using different `property` parameters, you can query node information in different ways, such as getting the edges and neighboring node values. Examples below show how to request this information for one or multiple nodes. | @@ -99,7 +99,7 @@ The response looks like: | Name | Type | Description | | --------- | ------ | ---------------------------------------------------------------------------- | | data | object | Data of the property label and value information, keyed by the queried nodes | -| nextToken | string | A token used to query [next page of data](/api/rest/v2/getting_started.html#pagination) | +| nextToken | string | A token used to query [next page of data](/api/rest/v2/index.html#pagination) | {: .doc-table} ## Examples diff --git a/api/rest/v2/observation.md b/api/rest/v2/observation.md index 84ae7fc23..a4d6fe638 100644 --- a/api/rest/v2/observation.md +++ b/api/rest/v2/observation.md @@ -2,7 +2,7 @@ layout: default title: Get statistical observations nav_order: 1 -parent: REST (v2) +parent: REST (V2) grand_parent: API published: true --- @@ -62,7 +62,7 @@ JSON data: | Name | Type | Description | |-------------------------------------------------------|--------|-----------------------------------------------------------------| -| key
Required | string | Your API key. See the section on [authentication](/api/rest/v2/getting_started.html#authentication) for details. | +| key
Required | string | Your API key. See the section on [authentication](/api/rest/v2/index.html#authentication) for details. | | date
Required | string | See [below](#date-string) for allowable values. | | variable.dcids
Required| list of strings | List of [DCIDs](/glossary.html#dcid) for the statistical variable to be queried. | | entity.dcids | list of strings | Comma-separated list of [DCIDs](/glossary.html#dcid) of entities to query. One of `entity.dcids` or `entity.expression` is required. Multiple `entity.dcids` parameters are allowed. | diff --git a/api/rest/v2/resolve.md b/api/rest/v2/resolve.md index c1fe8a034..4ea926124 100644 --- a/api/rest/v2/resolve.md +++ b/api/rest/v2/resolve.md @@ -2,7 +2,7 @@ layout: default title: Resolve entities nav_order: 4 -parent: REST (v2) +parent: REST (V2) grand_parent: API published: true --- @@ -73,7 +73,7 @@ JSON data: | Name | Type | Description | |---------------|-------|----------------| -| key
Required | string | Your API key. See the [section on authentication](/api/rest/v2/index.html#authentication) for a demo key, as well as instructions on how to get your own key. | +| key
Required | string | Your API key. See the [section on authentication](/api/rest/v2/index.html#authentication) for details. | | nodes
Required | list of strings | Comma-separated list of property values (e.g. entity name or DCID), or description of the node. This currently only supports the name of a place. | property
Required | string | [Relation expression](/api/rest/v2/#relation-expressions) that represents the relation of the given nodes to the queried entities. Note that this should always end with `->dcid` | diff --git a/api/rest/v2/sparql.md b/api/rest/v2/sparql.md index 3c086a35f..ab55f053d 100644 --- a/api/rest/v2/sparql.md +++ b/api/rest/v2/sparql.md @@ -2,7 +2,7 @@ layout: default title: Query with SPARQL nav_order: 5 -parent: REST (v2) +parent: REST (V2) grand_parent: API published: true --- @@ -38,7 +38,7 @@ JSON data: { "query": "SPARQL_QUERY" } | Name | Type | Description | | --------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| key
Required | string | Your API key. See the the section on [authentication](/api/rest/v2/getting_started.html#authentication) for instructions on how to get a key. | +| key
Required | string | Your API key. See the the section on [authentication](/api/rest/v2/index.html#authentication) for details. | | query
Required | string | A SPARQL query string.
In the query, all desired entities must be specified; wildcards are not supported. Each node or entity should have a `typeOf` condition, for example, ?ENTITY_NAME typeOf City. | {: .doc-table } diff --git a/api/rest/v2/troubleshooting.md b/api/rest/v2/troubleshooting.md index cdf369017..e61bde59b 100644 --- a/api/rest/v2/troubleshooting.md +++ b/api/rest/v2/troubleshooting.md @@ -2,7 +2,7 @@ layout: default title: Troubleshooting nav_order: 6 -parent: REST (v2) +parent: REST (V2) grand_parent: API published: true --- @@ -29,7 +29,7 @@ published: true } ``` -This is seen when your request is missing an API key. Please [request your own API key](/api/rest/v2/index.html#get-key). +This is seen when your request is missing an API key. Please [request your own API key](/api/index.html#get-key). ## Empty response diff --git a/api/sheets/index.md b/api/sheets/index.md index 3829e7729..a5f0f3641 100644 --- a/api/sheets/index.md +++ b/api/sheets/index.md @@ -1,7 +1,7 @@ --- layout: default title: Google Sheets -nav_order: 0 +nav_order: 20 parent: API has_children: true --- diff --git a/api/web_components/index.md b/api/web_components/index.md index 07b8956ac..80bb26223 100644 --- a/api/web_components/index.md +++ b/api/web_components/index.md @@ -2,7 +2,7 @@ layout: default title: Web Components parent: API -nav_order: 20 +nav_order: 30 has_children: true published: true --- diff --git a/contributing/index.md b/contributing/index.md index 68ee3c970..1854e0362 100644 --- a/contributing/index.md +++ b/contributing/index.md @@ -33,7 +33,7 @@ Data Commons welcomes the development of new tools that make the data on Data Co ### Share analysis -Example analyses are available on the [tutorials]](/tutorials) page. To add your analysis on that page, open a PR in [the Data Commons documentation repo](https://github.com/datacommonsorg/docsite/pulls){: target="_blank"} with the title of the analysis and links to its Colab notebook and Github raw .ipynb file. +Example analyses are available on the [tutorials](/api/python/tutorials.html) page. To add your analysis on that page, open a PR in [the Data Commons documentation repo](https://github.com/datacommonsorg/docsite/pulls){: target="_blank"} with the title of the analysis and links to its Colab notebook and Github raw .ipynb file. ### Update documentation diff --git a/custom_dc/quickstart.md b/custom_dc/quickstart.md index 7023513fc..d2e76b510 100644 --- a/custom_dc/quickstart.md +++ b/custom_dc/quickstart.md @@ -37,11 +37,14 @@ The "services" Docker container consists of the following Data Commons component - If you are developing on Windows, install [WSL 2](https://learn.microsoft.com/en-us/windows/wsl/install){: target="_blank"} (any distribution will do, but we recommend the default, Ubuntu), and enable [WSL 2 integration with Docker](https://docs.docker.com/desktop/wsl/){: target="_blank"}. - Install [Docker Desktop/Engine](https://docs.docker.com/engine/install/){: target="_blank"}. - Install [Git](https://git-scm.com/){: target="_blank"}. -- Get an API key to authorize requests from your site to the base Data Commons, by [filling out this form](https://docs.google.com/forms/d/e/1FAIpQLSeVCR95YOZ56ABsPwdH1tPAjjIeVDtisLF-8oDYlOxYmNZ7LQ/viewform?usp=dialog){: target="_blank"}. Typical turnaround times are 24-48 hours. - Optional: Get a [Github](http://github.com){: target="_blank"} account, if you would like to browse the Data Commons source repos using your browser. ## One-time setup steps {#setup} +### Get a Data Commons API key + +An API key is required to authorize requests from your site to the base Data Commons site. API keys are managed by a self-serve portal. To obtain an API key, go to [https://apikeys.datacommons.org](https://apikeys.datacommons.org){: target="_blank"} and request a key for the `api.datacommons.org` domain. + ### Enable Google Cloud APIs and get a Maps API key {#maps-key} 1. Go to [https://console.cloud.google.com/apis/dashboard](https://console.cloud.google.com/apis/dashboard){: target="_blank"} for your project.