TDL-18712 Add support of EU endpoint.

README.md

@@ -13,6 +13,9 @@ spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
 This tap:
 
 - Pulls raw data from the [Pendo API](https://developers.pendo.io/docs/?bash#overview).
+- Supports following two server
+  - US Server
+  - EU Server
 - Extracts the following resources:
   - Accounts
   - Features
@@ -37,7 +40,8 @@ This tap:
 
 **[accounts](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `account_id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `lastupdated`
@@ -48,31 +52,35 @@ This tap:
 
 **[features](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/feature)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `last_updated_at`
 - Transformations: Camel to snake case.
 
 **[guides](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `last_pdated_at`
 - Transformations: Camel to snake case.
 
 **[track types](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `last_pdated_at`
 - Transformations: Camel to snake case.
 
 **[visitors](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `lastupdated`
@@ -86,84 +94,96 @@ This tap:
 
 **[visitor_history](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `modified_ts` (Max from `ts` or `lastTs`)
 - Transformations: Camel to snake case.
 
 **[feature_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `day` or `hour`
 - Transformations: Camel to snake case.
 
 **[events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `day` or `hour`
 - Transformations: Camel to snake case.
 
 **[page_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `day` or `hour`
 - Transformations: Camel to snake case.
 
 **[guide_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `browserTime`
 - Transformations: Camel to snake case.
 
 **[poll_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `browserTime`
 - Transformations: Camel to snake case.
 
 **[track_events](https://developers.pendo.io/docs/?bash#get-an-account-by-id)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `visitor_id`, `account_id`, `server`, `remote_ip`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `day` or `hour`
 - Transformations: Camel to snake case.
 
 **[guides](https://developers.pendo.io/docs/?bash#entities)**
 
-- Endpoint: [https://api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- US Server Endpoint: [https://app.pendo.io/api/v1/aggregation](https://app.pendo.io/api/v1/aggregation)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/aggregation](https://app.eu.pendo.io/api/v1/aggregation)
 - Primary key fields: `id`
 - Replication strategy: INCREMENTAL (query filtered)
   - Bookmark: `last_pdated_at`
 - Transformations: Camel to snake case.
 
 **[metadata accounts](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)**
 
-- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
+- US Server Endpoint: [https://app.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/account)
 - Replication strategy: FULL_TABLE
 - Transformations: Camel to snake case.
 
 **[metadata visitors](https://developers.pendo.io/docs/?bash#automatically-generated-metadata)**
 
-- Endpoint: [https://api/v1/metadata/schema/account](https://app.pendo.io/api/v1/metadata/schema/visitor)
+- US Server Endpoint: [https://app.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor)
+- EU Server Endpoint: [https://app.eu.pendo.io/api/v1/metadata/schema/visitor](https://app.pendo.io/api/v1/metadata/schema/visitor)
 - Replication strategy: FULL_TABLE
 - Transformations: Camel to snake case.
 
 ## Authentication
 
 Authentication is managed by integration keys. An integration key may be created in the Pendo website: Settings -> Integrations -> Integration Keys.
 
+### Note 
+- If you selected eu_domain then please make sure you enter integration_key of Pendo EU subscription only.
 ## State
 
 ```json
@@ -246,6 +266,7 @@ Interrupted syncs for Event type stream are resumed via a bookmark placed during
    - `period` (string, `ABCdef123`): `dayRange` or `hourRange`
    - `lookback_window` (integer): 10 (For event objects. Default: 0)
    - `request_timeout` (integer): 300 (For passing timeout to the request. Default: 300)
+   - `eu_domain` (boolean, `true` or `false`): It is an optional property. EU domain refers to the physical storage location of an organization's data or information. Setting this config parameter to true ensures that it uses the EU endpoint to capture the records else it will use the US endpoint by default to capture the records.
 
    ```json
    {
@@ -254,7 +275,8 @@ Interrupted syncs for Event type stream are resumed via a bookmark placed during
      "period": "dayRange",
      "lookback_window": 10,
      "request_timeout": 300,
-     "include_anonymous_visitors": "true"
+     "include_anonymous_visitors": "true",
+     "eu_domain": "true"
    }
    ```
 

sample_config.json

@@ -1,4 +1,5 @@
 {
   "x_pendo_integration_key": "x_pendo_integration_key",
-  "start_date": "2017-01-01T00:00:00Z"
+  "start_date": "2017-01-01T00:00:00Z",
+  "eu_domain": "true"
 }

tap_pendo/streams.py

@@ -20,7 +20,8 @@
 from tap_pendo import utils as tap_pendo_utils
 
 KEY_PROPERTIES = ['id']
-BASE_URL = "https://app.pendo.io"
+US_BASE_URL = "https://app.pendo.io"
+EU_BASE_URL = "https://app.eu.pendo.io"
 
 LOGGER = singer.get_logger()
 session = requests.Session()
@@ -92,11 +93,16 @@ def __init__(self, endpoint, method, headers=None, params=None):
         self.headers = headers
         self.params = params
 
-    def get_url(self, **kwargs):
+    def get_url(self, is_eu_domain, **kwargs):
         """
         Concatenate  and format the dynamic values to the BASE_URL
         """
-        return BASE_URL + self.endpoint.format(**kwargs)
+        # Update url if `eu` domain is selected
+        if str(is_eu_domain).lower() == "true":
+            return EU_BASE_URL + self.endpoint.format(**kwargs)
+        else:
+            return US_BASE_URL + self.endpoint.format(**kwargs)
+
 
 
 class Stream():
@@ -156,7 +162,7 @@ def request(self, endpoint, params=None, **kwargs):
         }
 
         request_kwargs = {
-            'url': self.endpoint.get_url(**kwargs),
+            'url': self.endpoint.get_url(self.config.get('eu_domain', False), **kwargs),
             'method': self.endpoint.method,
             'headers': headers,
             'params': params

tests/unittests/test_endpoints_dict.py

@@ -14,5 +14,12 @@ def test_correct_values_passed_in_endpoint_object():
 def test_correct_endpoint_url():
     stream.endpoint = Endpoints(
         "/api/v1/visitor/{visitorID}/history", "GET")
-    formatted_url = stream.endpoint.get_url(visitorID='abc')
-    assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history'
+    formatted_url = stream.endpoint.get_url(visitorID='abc', is_eu_domain=False) # Assign is_eu_domain=False for US region endpoint
+    assert formatted_url == 'https://app.pendo.io/api/v1/visitor/abc/history'
+
+def test_eu_endpoint_url():
+    stream.endpoint = Endpoints(
+        "/api/v1/visitor/{visitorID}/history", "GET")
+    formatted_url = stream.endpoint.get_url(visitorID='abc', is_eu_domain=True)
+    # Verify formatted_url for eu endpoint
+    assert formatted_url == 'https://app.eu.pendo.io/api/v1/visitor/abc/history'