Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Union schema compatibility #28

Merged
merged 19 commits into from
Oct 12, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .buildkite/hooks/pre-command
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,5 @@ export CI_SNOWFLAKE_DBT_USER=$(gcloud secrets versions access latest --secret="C
export CI_SNOWFLAKE_DBT_WAREHOUSE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_WAREHOUSE" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HOST" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_CATALOG=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_CATALOG" --project="dbt-package-testing-363917")
1 change: 1 addition & 0 deletions .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,5 +69,6 @@ steps:
- "CI_DATABRICKS_DBT_HOST"
- "CI_DATABRICKS_DBT_HTTP_PATH"
- "CI_DATABRICKS_DBT_TOKEN"
- "CI_DATABRICKS_DBT_CATALOG"
commands: |
bash .buildkite/scripts/run_models.sh databricks
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
# dbt_pinterest_source v0.10.0
[PR #28](https://github.com/fivetran/dbt_pinterest_source/pull/28) includes the following updates:
## Feature update 🎉
- Unioning capability! This adds the ability to union source data from multiple pinterest connectors. Refer to the [README](https://github.com/fivetran/dbt_pinterest_source/blob/main/README.md) for more details.
fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved

## Under the hood 🚘
- Updated tmp models to union source data using the `fivetran_utils.union_data` macro.
- To distinguish which source each field comes from, added `source_relation` column in each staging model and applied the `fivetran_utils.source_relation` macro.
- Updated tests to account for the new `source_relation` column.

# dbt_pinterest_source v0.9.0
# Pinterest Ads v5 Upgrade
## 🚨 Breaking Changes 🚨:
Expand Down
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ If you are **not** using the [Pinterest transformation package](https://github.
```yaml
packages:
- package: fivetran/pinterest_source
version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
version: [">=0.10.0", "<0.11.0"] # we recommend using ranges to capture non-breaking changes automatically
```
## Step 3: Define database and schema variables
By default, this package runs using your destination and the `pinterest` schema. If this is not where your Pinterest Ads data is (for example, if your pinterest schema is named `pinterest_fivetran`), add the following configuration to your root `dbt_project.yml` file:
Expand All @@ -60,6 +60,18 @@ vars:
## (Optional) Step 5: Additional configurations
<details><summary>Expand for configurations</summary>

### Union multiple connectors
If you have multiple pinterest ads connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `pinterest_ads_union_schemas` OR `pinterest_ads_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:

```yml
vars:
pinterest_ads_union_schemas: ['pinterest_usa','pinterest_canada'] # use this if the data is in different schemas/datasets of the same database/project
pinterest_ads_union_databases: ['pinterest_usa','pinterest_canada'] # use this if the data is in different databases/projects but uses the same schema name
```
Please be aware that the native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.

To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.

### Passing Through Additional Metrics
By default, this package will select `clicks`, `impressions`, and `cost` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) if desired, but not required. Use the below format for declaring the respective pass-through variables:

Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'pinterest_source'
version: '0.9.0'
version: '0.10.0'
config-version: 2
require-dbt-version: [">=1.3.0", "<2.0.0"]
vars:
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json

Large diffs are not rendered by default.

14 changes: 7 additions & 7 deletions integration_tests/ci/sample.profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ integration_tests:
pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
port: 5439
schema: pinterest_source_integration_tests
schema: pinterest_source_integration_tests_3
threads: 8
bigquery:
type: bigquery
method: service-account-json
project: 'dbt-package-testing'
schema: pinterest_source_integration_tests
schema: pinterest_source_integration_tests_3
threads: 8
keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
snowflake:
Expand All @@ -33,7 +33,7 @@ integration_tests:
role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
schema: pinterest_source_integration_tests
schema: pinterest_source_integration_tests_3
threads: 8
postgres:
type: postgres
Expand All @@ -42,13 +42,13 @@ integration_tests:
pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
port: 5432
schema: pinterest_source_integration_tests
schema: pinterest_source_integration_tests_3
threads: 8
databricks:
catalog: null
catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
schema: pinterest_source_integration_tests
threads: 2
schema: pinterest_source_integration_tests_3
threads: 8
token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
type: databricks
9 changes: 7 additions & 2 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
name: 'pinterest_source_integration_tests'
version: '0.9.0'
version: '0.10.0'
profile: 'integration_tests'
config-version: 2

vars:
pinterest_schema: pinterest_source_integration_tests
pinterest_schema: pinterest_source_integration_tests_4
pinterest_ad_group_history_identifier: "pinterest_ad_group_history_data"
pinterest_campaign_history_identifier: "pinterest_campaign_history_data"
pinterest_pin_promotion_history_identifier: "pinterest_pin_promotion_history_data"
Expand All @@ -16,6 +16,11 @@ vars:
pinterest_advertiser_history_identifier: "pinterest_advertiser_history_data"
pinterest_advertiser_report_identifier: "pinterest_advertiser_report_data"

models:
pinterest_source:
tmp:
+materialized: table

seeds:
+quote_columns: "{{ true if target.type in ('redshift','postgres') else false }}"
pinterest_source_integration_tests:
Expand Down
4 changes: 3 additions & 1 deletion models/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,6 @@

{% docs impression_1 %} The number of paid pin impressions that occurred on the day of the record. {% enddocs %}

{% docs impression_2 %} The number of earned pin impressions that occurred on the day of the record. {% enddocs %}
{% docs impression_2 %} The number of earned pin impressions that occurred on the day of the record. {% enddocs %}

{% docs source_relation %}The source of the record if the unioning functionality is being used. If not this field will be empty.{% enddocs %}
2 changes: 1 addition & 1 deletion models/src_pinterest_ads.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
version: 2

sources:
- name: pinterest_ads
- name: pinterest_ads # This source will only be used if you are using a single microsoft_ads source connector. If multiple sources are being unioned, their tables will be directly referenced via adapter.get_relation.
fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved
schema: "{{ var('pinterest_schema', 'pinterest_ads') }}"
database: "{% if target.type != 'spark'%}{{ var('pinterest_database', target.database) }}{% endif %}"

Expand Down
32 changes: 31 additions & 1 deletion models/stg_pinterest_ads.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- ad_group_id
- _fivetran_synced
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: ad_group_id
description: Ad group ID.
tests:
Expand Down Expand Up @@ -43,9 +46,12 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- campaign_id
- _fivetran_synced
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: campaign_id
description: "{{ doc('campaign_id') }}"
tests:
Expand Down Expand Up @@ -76,9 +82,12 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- _fivetran_synced
- pin_promotion_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: pin_promotion_id
description: Pin promotion ID.
tests:
Expand Down Expand Up @@ -128,12 +137,15 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- date_day
- pin_promotion_id
- ad_group_id
- campaign_id
- advertiser_id
columns:
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: date_day
description: "{{ doc('date_day') }}"
tests:
Expand All @@ -160,11 +172,14 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- date_day
- ad_group_id
- campaign_id
- advertiser_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: date_day
description: "{{ doc('date_day') }}"
tests:
Expand Down Expand Up @@ -193,9 +208,12 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- updated_at
- advertiser_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: advertiser_id
description: "{{ doc('advertiser_id') }}"
tests:
Expand Down Expand Up @@ -224,9 +242,12 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- date_day
- advertiser_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: date_day
description: "{{ doc('date_day') }}"
tests:
Expand All @@ -247,10 +268,13 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- date_day
- campaign_id
- advertiser_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: date_day
description: "{{ doc('date_day') }}"
tests:
Expand All @@ -277,10 +301,13 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- keyword_id
- ad_group_id
- _fivetran_synced
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: keyword_id
description: Unique identifier of the keyword.
tests:
Expand Down Expand Up @@ -313,13 +340,16 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- source_relation
- date_day
- keyword_id
- pin_promotion_id
- ad_group_id
- campaign_id
- advertiser_id
columns:
- name: source_relation
description: "{{ doc('source_relation') }}"
- name: date_day
description: "{{ doc('date_day') }}"
tests:
Expand Down
9 changes: 8 additions & 1 deletion models/stg_pinterest_ads__ad_group_history.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,19 @@ fields as (
staging_columns=get_ad_group_history_columns()
)
}}

{{ fivetran_utils.source_relation(
union_schema_variable='pinterest_ads_union_schemas',
union_database_variable='pinterest_ads_union_databases')
}}

from base
),

final as (

select
source_relation,
id as ad_group_id,
name as ad_group_name,
status as ad_group_status,
Expand All @@ -33,7 +40,7 @@ final as (
placement_group,
start_time,
summary_status,
row_number() over (partition by id order by _fivetran_synced desc) = 1 as is_most_recent_record
row_number() over (partition by source_relation, id order by _fivetran_synced desc) = 1 as is_most_recent_record
from fields
)

Expand Down
11 changes: 9 additions & 2 deletions models/stg_pinterest_ads__ad_group_report.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,19 @@ fields as (
staging_columns=get_ad_group_report_columns()
)
}}

{{ fivetran_utils.source_relation(
union_schema_variable='pinterest_ads_union_schemas',
union_database_variable='pinterest_ads_union_databases')
}}

from base
),

final as (

select

select
source_relation,
{{ dbt.date_trunc('day', 'date') }} as date_day,
ad_group_id,
ad_group_name,
Expand Down
13 changes: 10 additions & 3 deletions models/stg_pinterest_ads__advertiser_history.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,19 @@ fields as (
staging_columns=get_advertiser_history_columns()
)
}}

{{ fivetran_utils.source_relation(
union_schema_variable='pinterest_ads_union_schemas',
union_database_variable='pinterest_ads_union_databases')
}}

from base
),

final as (

select

select
source_relation,
id as advertiser_id,
name as advertiser_name,
country,
Expand All @@ -30,7 +37,7 @@ final as (
owner_username,
advertiser_permissions, -- permissions was renamed in macro
updated_time as updated_at,
row_number() over (partition by id order by updated_time desc) = 1 as is_most_recent_record
row_number() over (partition by source_relation, id order by updated_time desc) = 1 as is_most_recent_record
from fields
)

Expand Down
11 changes: 9 additions & 2 deletions models/stg_pinterest_ads__advertiser_report.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,19 @@ fields as (
staging_columns=get_advertiser_report_columns()
)
}}

{{ fivetran_utils.source_relation(
union_schema_variable='pinterest_ads_union_schemas',
union_database_variable='pinterest_ads_union_databases')
}}

from base
),

final as (

select

select
source_relation,
{{ dbt.date_trunc('day', 'date') }} as date_day,
advertiser_id,
coalesce(impression_1,0) + coalesce(impression_2,0) as impressions,
Expand Down
Loading