Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-18712 Add support of EU endpoint. #95

Merged
merged 5 commits into from
Apr 25, 2022
Merged

TDL-18712 Add support of EU endpoint. #95

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
84534b7
Added support of eu endpoint.
prijendev Apr 22, 2022
4221978
Updated readme file
prijendev Apr 22, 2022
0652028
Updated readme file
prijendev Apr 22, 2022
3a9b66f
Updated readme for eu_domain property
prijendev Apr 22, 2022
7574737
Updated eu endpoint support logic
prijendev Apr 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 34 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
This tap:

- Pulls raw data from the [Pendo API](https://developers.pendo.io/docs/?bash#overview).
- Supports following two subscription
- US Subscription
- EU Subscription
- Extracts the following resources:
- Accounts
- Features
Expand All @@ -37,7 +40,8 @@ This tap:

**[accounts](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `account_id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `lastupdated`
Expand All @@ -48,31 +52,35 @@ This tap:

**[features](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/feature)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `last_updated_at`
- Transformations: Camel to snake case.

**[guides](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `last_pdated_at`
- Transformations: Camel to snake case.

**[track types](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `last_pdated_at`
- Transformations: Camel to snake case.

**[visitors](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `lastupdated`
Expand All @@ -86,77 +94,87 @@ This tap:

**[visitor_history](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `modified_ts` (Max from `ts` or `lastTs`)
- Transformations: Camel to snake case.

**[feature_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `day` or `hour`
- Transformations: Camel to snake case.

**[events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `day` or `hour`
- Transformations: Camel to snake case.

**[page_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `day` or `hour`
- Transformations: Camel to snake case.

**[guide_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `browserTime`
- Transformations: Camel to snake case.

**[poll_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `browserTime`
- Transformations: Camel to snake case.

**[track_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `day` or `hour`
- Transformations: Camel to snake case.

**[guides](https://developers.pendo.io/docs/?bash#entities)**

- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
- Primary key fields: `id`
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark: `last_pdated_at`
- Transformations: Camel to snake case.

**[metadata accounts](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)**

- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
- Replication strategy: FULL_TABLE
- Transformations: Camel to snake case.

**[metadata visitors](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)**

- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/visitor)
- US Subscription Endpoint: [https://app.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor)
- EU Subscription Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor)
- Replication strategy: FULL_TABLE
- Transformations: Camel to snake case.

Expand Down Expand Up @@ -358,4 +376,4 @@ Interrupted syncs for Event type stream are resumed via a bookmark placed during

---

Copyright © 2020 Stitch
Copyright © 2020 Stitch
14 changes: 10 additions & 4 deletions tap_pendo/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
from tap_pendo import utils as tap_pendo_utils

KEY_PROPERTIES = ['id']
BASE_URL = "https://app.pendo.io"
US_BASE_URL = "https://app.pendo.io"
EU_BASE_URL = "https://app.eu.pendo.io"

LOGGER = singer.get_logger()
session = requests.Session()
Expand Down Expand Up @@ -92,11 +93,16 @@ def __init__(self, endpoint, method, headers=None, params=None):
self.headers = headers
self.params = params

def get_url(self, **kwargs):
def get_url(self, integration_key, **kwargs):
"""
Concatenate and format the dynamic values to the BASE_URL
"""
return BASE_URL + self.endpoint.format(**kwargs)
# Update url if integration_key ends with `.eu`.
if integration_key[-3:] == ".eu":
return EU_BASE_URL + self.endpoint.format(**kwargs)
else:
return US_BASE_URL + self.endpoint.format(**kwargs)



class Stream():
Expand Down Expand Up @@ -156,7 +162,7 @@ def request(self, endpoint, params=None, **kwargs):
}

request_kwargs = {
'url': self.endpoint.get_url(**kwargs),
'url': self.endpoint.get_url(self.config['x_pendo_integration_key'], **kwargs),
'method': self.endpoint.method,
'headers': headers,
'params': params
Expand Down
11 changes: 9 additions & 2 deletions tests/unittests/test_endpoints_dict.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,12 @@ def test_correct_values_passed_in_endpoint_object():
def test_correct_endpoint_url():
stream.endpoint = Endpoints(
"/api/v1/visitor/{visitorID}/history", "GET")
formatted_url = stream.endpoint.get_url(visitorID='abc')
assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history'
formatted_url = stream.endpoint.get_url(visitorID='abc', integration_key="rf4007a20459.us") # Pass integration_key of US subscription account
assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history'

def test_eu_endpoint_url():
stream.endpoint = Endpoints(
"/api/v1/visitor/{visitorID}/history", "GET")
formatted_url = stream.endpoint.get_url(visitorID='abc', integration_key="rf4007a20459.eu")
# Verify formatted_url for eu subscription account by integration_key.
assert formatted_url == 'https://app.eu.pendo.io/api/v1/visitor/abc/history'