Feature/conversions

CHANGELOG.md

@@ -1,3 +1,28 @@
+# dbt_amazon_ads_source v0.4.0
+[PR #21](https://github.com/fivetran/dbt_amazon_ads_source/pull/21) includes the following updates:
+
+## Feature Update: Conversion support
+- We have added conversion metrics by default to the following staging models:
+  - `stg_amazon_ads__ad_group_level_report`
+  - `stg_amazon_ads__advertised_product_report`
+  - `stg_amazon_ads__campaign_level_report`
+  - `stg_amazon_ads__targeting_keyword_report`
+  - `stg_amazon_ads__search_term_ad_keyword_report`
+
+- The conversion metrics are the following:
+  - `purchases_30_d`: Number of attributed conversion events occurring within 30 days of an ad click.
+  - `sales_30_d`: Total value of sales occurring within 30 days of an ad click.
+- To bring in other conversion fields (`purchases_same_sku_30_d`, `sales_14_d`, etc.), please refer to our [passthrough column variables](https://github.com/fivetran/dbt_amazon_ads_source?tab=readme-ov-file#passing-through-additional-metrics).
+
+## Under the hood: Backwards compatibility
+- In the event that you were already passing the above fields in via our [passthrough columns](https://github.com/fivetran/dbt_amazon_ads_source?tab=readme-ov-file#passing-through-additional-metrics), the package will dynamically avoid "duplicate column" errors.
+- This was done via the new `amazon_ads_fill_pass_through_columns` and `amazon_ads_add_pass_through_columns` macros to ensure that the new conversion fields are backwards compatible with users who have already included them via passthrough fields.
+
+> The above new field additions are **breaking changes** for users who were not already bringing in conversion fields via passthrough columns.
+
+## Contributors
+- [Seer Interactive](https://www.seerinteractive.com/?utm_campaign=Fivetran%20%7C%20Models&utm_source=Fivetran&utm_medium=Fivetran%20Documentation)
+
 # dbt_amazon_ads_source v0.3.0
 [PR #17](https://github.com/fivetran/dbt_amazon_ads_source/pull/17) includes the following updates:
 ## Feature update 🎉

README.md

@@ -26,13 +26,13 @@ To use this dbt package, you must have the following:
 - At least one Fivetran Amazon_ads connector syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
-### Step 2: Install the package (skip if also using the `amazon_ads` transformation package)
-If you  are **not** using the [Amazon Ads transformation package](https://github.com/fivetran/dbt_amazon_ads), include the following `amazon_ads_source` package version in your `packages.yml` file.
+### Step 2: Install the package (skip if also using the `amazon_ads` transformation package or `ad_reporting` combo package)
+If you  are **not** using the [Amazon Ads transformation package](https://github.com/fivetran/dbt_amazon_ads) or the [Ad Reporting combination package](https://github.com/fivetran/dbt_ad_reporting), include the following `amazon_ads_source` package version in your `packages.yml` file.
 > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read dbt's Package Management documentation](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 ```yaml
 packages:
   - package: fivetran/amazon_ads_source
-    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ### Step 3: Define database and schema variables
@@ -53,6 +53,8 @@ vars:
 ```
 
 ### (Optional) Step 5: Additional configurations
+<details open><summary>Expand/Collapse details</summary>
+
 #### Union multiple connectors
 If you have multiple amazon_ads connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `amazon_ads_union_schemas` OR `amazon_ads_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
 
@@ -66,9 +68,9 @@ vars:
 To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
 
 #### Passing Through Additional Metrics
-By default, this package will select `clicks`, `impressions`, and `cost` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the following configurations to your `dbt_project.yml` file. These variables allow the pass-through fields to be aliased (`alias`) if desired, but not required. Use the following format for declaring the respective pass-through variables:
+By default, this package will select `clicks`, `impressions`, `cost`, `purchases_30_d`, and `sales_30_d` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the following configurations to your `dbt_project.yml` file. These variables allow the pass-through fields to be aliased (`alias`) and transformed (`transform_sql`) if desired, but not required. Use the following format for declaring the respective pass-through variables:
 
-> **Note** Ensure you exercised due diligence when adding metrics to these models. The metrics added by default (clicks, impressions, and cost) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example, metric averages, which may be inaccurately represented at the grain for reports created in this package. You want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+> **Note** Ensure you exercised due diligence when adding metrics to these models. The metrics added by default (clicks, impressions, cost, purchases, and sales amount) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example, metric averages, which may be inaccurately represented at the grain for reports created in this package. You want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
 
 ```yml
 vars:
@@ -77,10 +79,11 @@ vars:
         alias: "custom_field"
     amazon_ads__ad_group_passthrough_metrics:
       - name: "unique_string_field"
-        alias: "field_id"
+        transform_sql: "coalesce(unique_string_field, 'NA')"
     amazon_ads__advertised_product_passthrough_metrics: 
       - name: "new_custom_field"
         alias: "custom_field"
+        transform_sql: "coalesce(custom_field, 'NA')" # reference alias in transform_sql if aliasing 
       - name: "a_second_field"
     amazon_ads__targeting_keyword_passthrough_metrics:
       - name: "this_field"
@@ -90,7 +93,7 @@ vars:
 ```
 
 #### Changing the Build Schema
-By default, this package will build the Amazon_ads staging models within a schema titled (<target_schema> + `amazon_ads_source`) in your destination. If this is not where you would like your Amazon Ads staging data to be written, add the following configuration to your root `dbt_project.yml` file:
+By default, this package will build the Amazon_ads staging models (11 views, 11 tables) within a schema titled (<target_schema> + `amazon_ads_source`) in your destination. If this is not where you would like your Amazon Ads staging data to be written, add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 models:
@@ -99,7 +102,7 @@ models:
 ```
 
 #### Change the source table references
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable. (This is not available when running the package on multiple unioned connectors):
 
 > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
@@ -108,6 +111,8 @@ vars:
     amazon_ads_<default_source_table_name>_identifier: your_table_name 
 ```
 
+</details>
+
 ### (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand for more details</summary>
 
@@ -133,7 +138,12 @@ The Fivetran team maintaining this package _only_ maintains the latest version o
 ### Contributions
 A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions.
 
-We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) to learn how to contribute to a dbt package.
+We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package.
+
+#### Contributors
+We thank [everyone](https://github.com/fivetran/amazon_ads_source/graphs/contributors) who has taken the time to contribute. Each PR, bug report, and feature request has made this package better and is truly appreciated.
+
+A special thank you to [Seer Interactive](https://www.seerinteractive.com/?utm_campaign=Fivetran%20%7C%20Models&utm_source=Fivetran&utm_medium=Fivetran%20Documentation), who we closely collaborated with to introduce native conversion support to our Ad packages.
 
 ## Are there any resources available?
 - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_amazon_ads_source/issues/new/choose) section to find the right avenue of support for you.

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'amazon_ads_source'
-version: '0.3.0'
+version: '0.4.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: amazon_ads_source_integration_tests_02
+      schema: amazon_ads_source_integration_tests_03
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: amazon_ads_source_integration_tests_02
+      schema: amazon_ads_source_integration_tests_03
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: amazon_ads_source_integration_tests_02
+      schema: amazon_ads_source_integration_tests_03
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: amazon_ads_source_integration_tests_02
+      schema: amazon_ads_source_integration_tests_03
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: amazon_ads_source_integration_tests_02
+      schema: amazon_ads_source_integration_tests_03
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,13 +1,13 @@
 config-version: 2
 
 name: 'amazon_ads_source_integration_tests'
-version: '0.3.0'
+version: '0.4.0'
 
 profile: 'integration_tests' 
 
 vars:
   amazon_ads_source:
-    amazon_ads_schema: amazon_ads_source_integration_tests_02
+    amazon_ads_schema: amazon_ads_source_integration_tests_03
     amazon_ads_ad_group_history_identifier: "ad_group_history_data"
     amazon_ads_ad_group_level_report_identifier: "ad_group_level_report_data"
     amazon_ads_advertised_product_report_identifier: "advertised_product_report_data"
@@ -20,6 +20,12 @@ vars:
     amazon_ads_targeting_keyword_report_identifier: "targeting_keyword_report_data"
     amazon_ads_search_term_ad_keyword_report_identifier: "search_term_ad_keyword_report_data"
 
+  amazon_ads__campaign_passthrough_metrics:
+      - name: sales_7_d
+      - name: purchases_30_d 
+        alias: purchases_alias
+      - name: purchases_14_d
+
 dispatch:
   - macro_namespace: dbt_utils
     search_order: ['spark_utils', 'dbt_utils']
@@ -31,8 +37,4 @@ seeds:
       _fivetran_synced: "timestamp"
       campaign_budget_amount: "float"
       click_through_rate: "float"
-      keyword_bid: "float"
-
-dispatch:
-  - macro_namespace: dbt_utils
-    search_order: ['spark_utils', 'dbt_utils']
+      keyword_bid: "float"

integration_tests/requirements.txt

@@ -4,4 +4,4 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
+dbt-databricks>=1.6.0,<2.0.0

integration_tests/seeds/campaign_level_report_data.csv

@@ -1,11 +1,11 @@
-campaign_id,date,_fivetran_synced,campaign_applicable_budget_rule_id,campaign_applicable_budget_rule_name,campaign_bidding_strategy,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions,campaign_rule_based_budget_amount
-2187,2022-07-26,2022-10-14 11:54:40.980000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-07-21,2022-10-14 11:52:57.796000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-08-29,2022-10-14 11:40:45.808000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-08-31,2022-10-14 11:42:29.065000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-07-27,2022-10-14 11:54:40.980000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-08-28,2022-10-14 11:40:45.807000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-09-06,2022-10-14 11:42:29.062000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-07-11,2022-10-14 12:07:12.424000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET0.0,2,1.67,1095,
-2187,2022-07-15,2022-10-14 12:07:12.428000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
-2187,2022-07-18,2022-10-14 11:52:57.802000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,
+campaign_id,date,_fivetran_synced,campaign_applicable_budget_rule_id,campaign_applicable_budget_rule_name,campaign_bidding_strategy,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions,campaign_rule_based_budget_amount,sales_7_d,purchases_30_d,sales_30_d
+2187,2022-07-26,2022-10-14 11:54:40.980000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,12,44,54
+2187,2022-07-21,2022-10-14 11:52:57.796000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,2,34,345
+2187,2022-08-29,2022-10-14 11:40:45.808000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,,,
+2187,2022-08-31,2022-10-14 11:42:29.065000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,,,
+2187,2022-07-27,2022-10-14 11:54:40.980000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,3,35,350
+2187,2022-08-28,2022-10-14 11:40:45.807000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,,,
+2187,2022-09-06,2022-10-14 11:42:29.062000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,4,24,335
+2187,2022-07-11,2022-10-14 12:07:12.424000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET0.0,2,1.67,1095,,0,64,234
+2187,2022-07-15,2022-10-14 12:07:12.428000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,,,
+2187,2022-07-18,2022-10-14 11:52:57.802000,,,optimizeForSales,2000.0,USD,DAILY_BUDGET,0,0.0,,,,,

macros/amazon_ads_add_pass_though_columns.sql

@@ -0,0 +1,28 @@
+{# Backwards compatible version of fivetran_utils.add_pass_through_columns #}
+
+{% macro amazon_ads_add_pass_through_columns(base_columns, pass_through_fields, except_fields=[]) %}
+
+{% if pass_through_fields %}
+    {% for column in pass_through_fields %}
+
+        {% if column is mapping %}
+        {% set col_name = column.alias|default(column.name)|lower %}
+
+            {% if col_name not in except_fields %}
+                {% if column.alias %}
+                    {% do base_columns.append({ "name": column.name, "alias": column.alias, "datatype": column.datatype if column.datatype else dbt.type_string()}) %}
+                {% else %}
+                    {% do base_columns.append({ "name": column.name, "datatype": column.datatype if column.datatype else dbt.type_string()}) %}
+                {% endif %}
+            {% endif %}
+
+        {% else %}
+            {% if column|lower not in except_fields %}
+                {% do base_columns.append({ "name": column, "datatype": dbt.type_string()}) %}
+            {% endif %}
+        {% endif %}
+
+    {% endfor %}
+{% endif %}
+
+{% endmacro %}

macros/amazon_ads_fill_pass_through_columns.sql

@@ -0,0 +1,17 @@
+{# Backwards compatible version of fivetran_utils.fill_pass_through_columns #}
+
+{% macro amazon_ads_fill_pass_through_columns(pass_through_fields, except=[]) %}
+
+{% if pass_through_fields %}
+    {% for field in pass_through_fields %}
+        {% if (field.alias if field.alias else field.name) not in except %}
+            {% if field.transform_sql %}
+                , coalesce(cast({{ field.transform_sql }} as {{ dbt.type_float() }}), 0) as {{ field.alias if field.alias else field.name }}
+            {% else %}
+                , coalesce(cast({{ field.alias if field.alias else field.name }} as {{ dbt.type_float() }}), 0) as {{ field.alias if field.alias else field.name }}
+            {% endif %}
+        {% endif %}
+    {% endfor %}
+{% endif %}
+
+{% endmacro %}

macros/get_ad_group_level_report_columns.sql

@@ -7,9 +7,12 @@
     {"name": "cost", "datatype": dbt.type_float()},
     {"name": "date", "datatype": "date"},
     {"name": "impressions", "datatype": dbt.type_int()},
+    {"name": "purchases_30_d", "datatype": dbt.type_int()},
+    {"name": "sales_30_d", "datatype": dbt.type_float()}
 ] %}
 
-{{ fivetran_utils.add_pass_through_columns(columns, var('amazon_ads__ad_group_passthrough_metrics')) }}
+{# Add backwards compatibility if conversion metrics were added via passthrough columns prior to them being brought in by default #}
+{{ amazon_ads_add_pass_through_columns(base_columns=columns, pass_through_fields=var('amazon_ads__ad_group_passthrough_metrics'), except_fields=['purchases_30_d', 'sales_30_d']) }}
 
 {{ return(columns) }}
 

macros/get_advertised_product_report_columns.sql

@@ -13,9 +13,12 @@
     {"name": "cost", "datatype": dbt.type_float()},
     {"name": "date", "datatype": "date"},
     {"name": "impressions", "datatype": dbt.type_int()},
+    {"name": "purchases_30_d", "datatype": dbt.type_int()},
+    {"name": "sales_30_d", "datatype": dbt.type_float()}
 ] %}
 
-{{ fivetran_utils.add_pass_through_columns(columns, var('amazon_ads__advertised_product_passthrough_metrics')) }}
+{# Add backwards compatibility if conversion metrics were added via passthrough columns prior to them being brought in by default #}
+{{ amazon_ads_add_pass_through_columns(base_columns=columns, pass_through_fields=var('amazon_ads__advertised_product_passthrough_metrics'), except_fields=['purchases_30_d', 'sales_30_d']) }}
 
 {{ return(columns) }}