diff --git a/README.md b/README.md index b8cc15c..94043fd 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,9 @@ spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md). This tap: - Pulls raw data from the [Pendo API](https://developers.pendo.io/docs/?bash#overview). +- Supports following two subscription + - US Subscription + - EU Subscription - Extracts the following resources: - Accounts - Features @@ -37,7 +40,8 @@ This tap: **[accounts](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `account_id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `lastupdated` @@ -48,7 +52,8 @@ This tap: **[features](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/feature) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `last_updated_at` @@ -56,7 +61,8 @@ This tap: **[guides](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `last_pdated_at` @@ -64,7 +70,8 @@ This tap: **[track types](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `last_pdated_at` @@ -72,7 +79,8 @@ This tap: **[visitors](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `lastupdated` @@ -86,7 +94,8 @@ This tap: **[visitor_history](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `modified_ts` (Max from `ts` or `lastTs`) @@ -94,7 +103,8 @@ This tap: **[feature_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `day` or `hour` @@ -102,7 +112,8 @@ This tap: **[events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `day` or `hour` @@ -110,7 +121,8 @@ This tap: **[page_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `day` or `hour` @@ -118,7 +130,8 @@ This tap: **[guide_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `browserTime` @@ -126,7 +139,8 @@ This tap: **[poll_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `browserTime` @@ -134,7 +148,8 @@ This tap: **[track_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `day` or `hour` @@ -142,7 +157,8 @@ This tap: **[guides](https://developers.pendo.io/docs/?bash#entities)** -- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation) - Primary key fields: `id` - Replication strategy: INCREMENTAL (query filtered) - Bookmark: `last_pdated_at` @@ -150,13 +166,15 @@ This tap: **[metadata accounts](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)** -- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account) - Replication strategy: FULL_TABLE - Transformations: Camel to snake case. **[metadata visitors](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)** -- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/visitor) +- US Subscription Endpoint: [https://app.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor) +- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor) - Replication strategy: FULL_TABLE - Transformations: Camel to snake case. @@ -358,4 +376,4 @@ Interrupted syncs for Event type stream are resumed via a bookmark placed during --- -Copyright © 2020 Stitch +Copyright © 2020 Stitch \ No newline at end of file diff --git a/tap_pendo/streams.py b/tap_pendo/streams.py index 8310058..e3d579e 100644 --- a/tap_pendo/streams.py +++ b/tap_pendo/streams.py @@ -20,7 +20,8 @@ from tap_pendo import utils as tap_pendo_utils KEY_PROPERTIES = ['id'] -BASE_URL = "https://app.pendo.io" +US_BASE_URL = "https://app.pendo.io" +EU_BASE_URL = "https://app.eu.pendo.io" LOGGER = singer.get_logger() session = requests.Session() @@ -92,11 +93,16 @@ def __init__(self, endpoint, method, headers=None, params=None): self.headers = headers self.params = params - def get_url(self, **kwargs): + def get_url(self, integration_key, **kwargs): """ Concatenate and format the dynamic values to the BASE_URL """ - return BASE_URL + self.endpoint.format(**kwargs) + # Update url if integration_key ends with `.eu`. + if integration_key[-3:] == ".eu": + return EU_BASE_URL + self.endpoint.format(**kwargs) + else: + return US_BASE_URL + self.endpoint.format(**kwargs) + class Stream(): @@ -156,7 +162,7 @@ def request(self, endpoint, params=None, **kwargs): } request_kwargs = { - 'url': self.endpoint.get_url(**kwargs), + 'url': self.endpoint.get_url(self.config['x_pendo_integration_key'], **kwargs), 'method': self.endpoint.method, 'headers': headers, 'params': params diff --git a/tests/unittests/test_endpoints_dict.py b/tests/unittests/test_endpoints_dict.py index 0068da6..3e03681 100644 --- a/tests/unittests/test_endpoints_dict.py +++ b/tests/unittests/test_endpoints_dict.py @@ -14,5 +14,12 @@ def test_correct_values_passed_in_endpoint_object(): def test_correct_endpoint_url(): stream.endpoint = Endpoints( "/api/v1/visitor/{visitorID}/history", "GET") - formatted_url = stream.endpoint.get_url(visitorID='abc') - assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history' \ No newline at end of file + formatted_url = stream.endpoint.get_url(visitorID='abc', integration_key="rf4007a20459.us") # Pass integration_key of US subscription account + assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history' + +def test_eu_endpoint_url(): + stream.endpoint = Endpoints( + "/api/v1/visitor/{visitorID}/history", "GET") + formatted_url = stream.endpoint.get_url(visitorID='abc', integration_key="rf4007a20459.eu") + # Verify formatted_url for eu subscription account by integration_key. + assert formatted_url == 'https://app.eu.pendo.io/api/v1/visitor/abc/history' \ No newline at end of file