Tiktok API version upgrade schema changes (#24)

* modify API endpoint * add fields in the campaigns schema * schema changes for adgroup * fix unit test * Schema changes for ads stream * advertisers schema change * update Advertisers key properties * update field types for insight/s streams * set the advertiser_id if it is not present in record * update placement field to placement_type * setup and changelog updates * add few more fields in the ads and adgroups stream * pass the advertiser_id in the param as json formatted string * fix failing unit test cases * update changelog * add new fields for the insight related streams * address review comments * add new field - rf_campaign_type in the schemas * remove aeo_type field * update primary key for advertisers stream in the base.py * modify all fields test * update docs links in readme * update field type for conversion rates * retry for code - 40100 * fix the integration tests * remove few fields from test all field test * Introduce new stream - campaign_insights_by_province (#25) * Add new stream - campaign_insights_by_province * update base.py with the new stream - campaign_insights_by_province * remove user_action field from the campaign insights stream * update the README.md file to include the latest stream details. * format the campaign insights json file * update field type for conversion rates * remove campaigns by province stream from start date and pagination
singer-io · Aug 3, 2023 · b854b6a · b854b6a
1 parent 003dc44
commit b854b6a
Show file tree

Hide file tree

Showing 27 changed files with 1,278 additions and 169 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,14 @@
 # Changelog
 
+## 1.0.0
+ * Tiktok API version upgrade from v1.2 to v1.3 [#24] (https://github.com/singer-io/tap-tiktok-ads/pull/24)
+    * Contains the schema changes as per the new API.
+    * Boolean Types were represented as 0/1 previously which will be changed to true/false.
+    * Removal of query parameters, changes in URL and response Structure.
+    * Authentication Endpoints upgrade.
+    * Removal and renaming of fields, update data type of fields.
+  * Introduced new feature to fetch the deleted records [#23] (https://github.com/singer-io/tap-tiktok-ads/pull/23)
+
 ## 0.4.0
  * Sync records from the start date specified in the config. [#22] (https://github.com/singer-io/tap-tiktok-ads/pull/22)
 

diff --git a/README.md b/README.md
@@ -7,16 +7,17 @@ spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
 This tap: 
 
 - Pulls raw data from the [TikTok Marketing API](https://ads.tiktok.com/marketing_api/docs). 
-- Extracts the following resources from TikTok Marketing API for a single repository: 
-	- [Accounts](https://ads.tiktok.com/marketing_api/docs?id=1708503235186689) - Endpoint: https://business-api.tiktok.com/open_api/v1.2/advertiser/info/ 
-	- [Campaigns](https://ads.tiktok.com/marketing_api/docs?id=1708582970809346) - Endpoint: https://ads.tiktok.com/open_api/v1.2/campaign/get/ 
-	- [Adgroups](https://ads.tiktok.com/marketing_api/docs?id=1708503489590273) - Endpoint: https://ads.tiktok.com/open_api/v1.2/adgroup/get/ 
-	- [Ads](https://ads.tiktok.com/marketing_api/docs?id=1708572923161602) - Endpoint: https://business-api.tiktok.com/open_api/v1.2/ad/get/ 
-	- [Reporting](https://ads.tiktok.com/marketing_api/docs?id=1701890949889025) - Endpoint: https://ads.tiktok.com/open_api/v1.2/reports/integrated/get/ 
-		- [Ad Insights](https://ads.tiktok.com/marketing_api/docs?id=1707957200780290)
-		- [Ad Insights by Age and Gender](https://ads.tiktok.com/marketing_api/docs?id=1707957217727489)
-		- [Ad Insights by Country](https://ads.tiktok.com/marketing_api/docs?id=1707957217727489)
-		- [Ad Insights by Platform](https://ads.tiktok.com/marketing_api/docs?id=1707957217727489)
+- Extracts the following resources from TikTok Marketing API for a single repository:
+	- [Accounts](https://ads.tiktok.com/marketing_api/docs?id=1739593083610113) - Endpoint: https://business-api.tiktok.com/open_api/v1.3/advertiser/info/ 
+	- [Campaigns](https://ads.tiktok.com/marketing_api/docs?id=1739315828649986) - Endpoint: https://ads.tiktok.com/open_api/v1.3/campaign/get/ 
+	- [Adgroups](https://ads.tiktok.com/marketing_api/docs?id=1739314558673922) - Endpoint: https://ads.tiktok.com/open_api/v1.3/adgroup/get/ 
+	- [Ads](https://ads.tiktok.com/marketing_api/docs?id=1735735588640770) - Endpoint: https://business-api.tiktok.com/open_api/v1.3/ad/get/ 
+	- [Reporting](https://ads.tiktok.com/marketing_api/docs?id=1751087777884161) - Endpoint: https://ads.tiktok.com/open_api/v1.3/report/integrated/get/ 
+		- [Ad Insights](https://ads.tiktok.com/marketing_api/docs?id=1738864915188737)
+		- [Ad Insights by Age and Gender](https://ads.tiktok.com/marketing_api/docs?id=1738864928947201)
+		- [Ad Insights by Country](https://ads.tiktok.com/marketing_api/docs?id=1738864928947201)
+		- [Ad Insights by Platform](https://ads.tiktok.com/marketing_api/docs?id=1738864928947201)
+        - [Campaign Insights by Province](https://ads.tiktok.com/marketing_api/docs?id=1738864928947201)
 - Outputs the schema for each resource
 - Incrementally pulls data based on the input state
 

diff --git a/sample_config.json b/sample_config.json
@@ -3,6 +3,7 @@
   "start_date": "2020-05-01T00:00:00Z",
   "user_agent": "user <email>",
   "accounts": "123456789, 123124142",
+  "include_deleted": "true",
   "sandbox": "true",
   "request_timeout": 300
 }
diff --git a/setup.py b/setup.py
@@ -3,7 +3,7 @@
 
 setup(
     name="tap-tiktok-ads",
-    version="0.4.0",
+    version="1.0.0",
     description="Singer.io tap for extracting data",
     author="Stitch",
     url="http://singer.io",

diff --git a/tap_tiktok_ads/__init__.py b/tap_tiktok_ads/__init__.py
@@ -6,7 +6,7 @@
 from tap_tiktok_ads.discover import discover
 from tap_tiktok_ads.sync import sync
 
-REQUIRED_CONFIG_KEYS = ['start_date', 'user_agent', 'access_token', 'accounts']
+REQUIRED_CONFIG_KEYS = ['start_date', 'access_token', 'accounts']
 LOGGER = singer.get_logger()
 
 
@@ -22,7 +22,8 @@ def main():
         raise Exception("Please provide atleast 1 Account ID.")
     try:
         # create list of accounts
-        accounts_list = [int(i) for i in accounts.split(",")]
+        # typecasting account to determine string is an integer
+        accounts_list = [str(int(account)) for account in accounts.split(",")]
     except ValueError:
         raise Exception("Provided list of account IDs contains invalid IDs. Kindly check your Account IDs.") from None
     # update string to list in the config
@@ -31,7 +32,6 @@ def main():
     with TikTokClient(access_token=args.config['access_token'],
                       advertiser_id=args.config['accounts'],
                       sandbox=args.config.get('sandbox', "false"),
-                      user_agent=args.config['user_agent'],
                       request_timeout=args.config.get('request_timeout')) as tik_tok_client:
 
         # If discover flag was passed, run discovery mode and dump output to stdout

diff --git a/tap_tiktok_ads/client.py b/tap_tiktok_ads/client.py
@@ -1,6 +1,6 @@
 from urllib.parse import urlencode
 import backoff
-
+import json
 import requests
 import backoff
 import singer
@@ -13,9 +13,7 @@
 REQUEST_TIMEOUT = 300
 
 LOGGER = singer.get_logger()
-
-ENDPOINT_BASE = "https://{api}.tiktok.com/open_api/v1.2"
-TOKEN_URL = 'https://{api}.tiktok.com/open_api/v1.2/user/info'
+ENDPOINT_VERSION = "v1.3"
 
 # pylint: disable=missing-class-docstring
 class TikTokAdsClientError(Exception):
@@ -30,7 +28,7 @@ def should_retry(e):
     error_code = response.json().get("code")
     # Backoff in case of below error codes. Refer doc: https://ads.tiktok.com/marketing_api/docs?rid=xmtaqatxqj8&id=1737172488964097
     # for more information.
-    if error_code in (40200, 40201, 40202, 40700, 50000, 50002):
+    if error_code in (40100, 40200, 40201, 40202, 40700, 50000, 50002):
         return True
     if (type(e) == Exception and type(e.args[0][1]) == ConnectionResetError) or type(e) == ConnectionResetError:
         # Tap raises Exception: ConnectionResetError(104, 'Connection reset by peer').
@@ -91,7 +89,7 @@ def check_access_token(self):
         headers['Access-Token'] = self.__access_token
         headers['Accept'] = 'application/json'
         response = self.__session.get(
-            url='https://{}.tiktok.com/open_api/v1.2/user/info'.format(self.__base_url_prefix),
+            url='https://{}.tiktok.com/open_api/{}/user/info'.format(self.__base_url_prefix, ENDPOINT_VERSION),
             headers=headers,
             timeout=self.__request_timeout)
 
@@ -110,7 +108,7 @@ def check_access_token(self):
                 "Access-Token": self.__access_token
             }
             params = {
-                "advertiser_ids": self.__advertiser_id
+                "advertiser_ids": json.dumps(self.__advertiser_id)
             }
             if self.__base_url_prefix == 'sandbox-ads':
                 return True
@@ -135,7 +133,7 @@ def request(self, method, url=None, path=None, **kwargs):
             self.__verified = self.check_access_token()
 
         if not url and self.__base_url is None:
-            self.__base_url = 'https://{}.tiktok.com/open_api/v1.2'.format(self.__base_url_prefix)
+            self.__base_url = 'https://{}.tiktok.com/open_api/{}'.format(self.__base_url_prefix, ENDPOINT_VERSION)
 
         if not url and path:
             url = f'{self.__base_url}/{path}'

diff --git a/tap_tiktok_ads/schemas/ad_insights.json b/tap_tiktok_ads/schemas/ad_insights.json
@@ -8,13 +8,13 @@
     "advertiser_id": {
       "type": [
         "null",
-        "integer"
+        "string"
       ]
     },
     "ad_id": {
       "type": [
         "null",
-        "integer"
+        "string"
       ]
     },
     "ad_name": {
@@ -39,7 +39,7 @@
     "adgroup_id": {
       "type": [
         "null",
-        "integer"
+        "string"
       ]
     },
     "adgroup_name": {
@@ -51,7 +51,7 @@
     "campaign_id": {
       "type": [
         "null",
-        "integer"
+        "string"
       ]
     },
     "campaign_name": {
@@ -60,7 +60,7 @@
         "string"
       ]
     },
-    "placement": {
+    "placement_type": {
       "type": [
         "null",
         "string"
@@ -341,6 +341,108 @@
         "null",
         "string"
       ]
+    },
+    "gross_impressions": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "is_smart_creative": {
+      "type": [
+        "null",
+        "boolean"
+      ]
+    },
+    "conversion_rate_v2": {
+      "type": [
+        "null",
+        "number"
+      ]
+    },
+    "real_time_conversion_rate_v2": {
+      "type": [
+        "null",
+        "number"
+      ]
+    },
+    "objective_type": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "app_promotion_type": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "split_test": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "campaign_budget": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "campaign_dedicate_type": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "opt_status": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "budget": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "smart_target": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "bid_strategy": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "bid": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "call_to_action": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "image_mode": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "billing_event": {
+      "type": [
+        "null",
+        "string"
+      ]
     }
   }
 }