Merge pull request #52 from fivetran/MagicBot/dbt-utils-cross-db-migr…

…ation

Updates for dbt-utils to dbt-core cross-db macro migration

.gitignore

@@ -2,5 +2,6 @@
 target/
 dbt_modules/
 logs/
+env/
 dbt_packages/
 .DS_Store

CHANGELOG.md

@@ -1,3 +1,45 @@
+# dbt_stripe_source v0.8.0
+
+## 🚨 Breaking Changes 🚨:
+[PR #52](https://github.com/fivetran/dbt_stripe_source/pull/52) includes the following breaking changes:
+- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically `{{ dbt_utils.<macro> }}` have been updated to `{{ dbt.<macro> }}` for the below macros:
+    - `any_value`
+    - `bool_or`
+    - `cast_bool_to_text`
+    - `concat`
+    - `date_trunc`
+    - `dateadd`
+    - `datediff`
+    - `escape_single_quotes`
+    - `except`
+    - `hash`
+    - `intersect`
+    - `last_day`
+    - `length`
+    - `listagg`
+    - `position`
+    - `replace`
+    - `right`
+    - `safe_cast`
+    - `split_part`
+    - `string_literal`
+    - `type_bigint`
+    - `type_float`
+    - `type_int`
+    - `type_numeric`
+    - `type_string`
+    - `type_timestamp`
+    - `array_append`
+    - `array_concat`
+    - `array_construct`
+- For `current_timestamp` and `current_timestamp_in_utc` macros, the dispatch AND the macro names have been updated to the below, respectively:
+    - `dbt.current_timestamp_backcompat`
+    - `dbt.current_timestamp_in_utc_backcompat`
+- `dbt_utils.surrogate_key` has also been updated to `dbt_utils.generate_surrogate_key`. Since the method for creating surrogate keys differ, we suggest all users do a `full-refresh` for the most accurate data. For more information, please refer to dbt-utils [release notes](https://github.com/dbt-labs/dbt-utils/releases) for this update.
+- Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
+- Updated README to include instructions on how to use metadata variable in cases of dictionary arguments. ([#57](https://github.com/fivetran/dbt_stripe_source/pull/57))
+
+
 # dbt_stripe_source v0.7.4
 ## Under the Hood
 - The `plan_interval` field within the `stg_stripe__plan` model has been aliased within the `get_plan_columns()` macro to allow for more flexibility across warehouses. ([#58](https://github.com/fivetran/dbt_stripe_source/pull/58))

README.md

@@ -3,7 +3,7 @@
         href="https://github.com/fivetran/dbt_stripe_source/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
     <a alt="dbt-core">
-        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.0.0_<2.0.0-orange.svg" /></a>
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_,<2.0.0-orange.svg" /></a>
     <a alt="Maintained?">
         <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
     <a alt="PRs">
@@ -39,7 +39,7 @@ Include the following stripe_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/stripe_source
-    version: [">=0.7.0", "<0.8.0"]
+    version: [">=0.8.0", "<0.9.0"]
 ```
 ## Step 3: Define database and schema variables
 By default, this package runs using your destination and the `stripe` schema. If this is not where your stripe data is (for example, if your stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -90,20 +90,46 @@ vars:
 ```
 
 ### Pivoting out Metadata Properties
-By default, this package selects the `metadata` JSON field within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. However, you likely have properties within the `metadata` JSON field you would like to pivot out and include in the respective downstream staging model.
+Oftentimes you may have custom fields within your source tables that is stored as a JSON object that you wish to pass through. By leveraging the `metadata` variable, this package pivot out fields into their own columns. The metadata variables accept dictionaries in addition to strings.
+
+Additionally, if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field, Below are examples of how you would add the respective fields.
+
+The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
+
+```yml
+vars: 
+  stripe__charge_metadata:
+    - name: metadata_field_1
+  stripe__invoice_metadata: 
+    - name: metadata_field_2
+  stripe__payment_intent_metadata:
+    - name: incompatible.field
+      alias: rename_incompatible_field
+  stripe__payment_method_metadata:
+    - name: field_is_reserved_word
+      alias: field_is_reserved_word_xyz
+  stripe__payout_metadata:
+    - name: 123
+      alias: one_two_three
+  stripe__plan_metadata:
+    - name: rename
+    - alias: renamed_field
+  stripe__refund_metadata:
+    - name: metadata_field_3
+    - name: metadata_field_4
+  stripe__subscription_metadata:
+    - name: metadata_field_5
+  stripe__customer_metadata:
+    - name: metadata_field_6
+
+```
+
+Alternatively, if you only have strings in your JSON object, the metadata variable accepts the following configuration as well.
+>**Note**: `stripe__plan_metadata` is only shown below, but the format will work for all metadata variables. 
 
-If there are properties in the `metadata` JSON field that you'd like to pivot out into columns, add the respective variable(s) to your root `dbt_project.yml` file:
 ```yml
 vars:
-    stripe__charge_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__invoice_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payment_intent_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payment_method_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payout_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
     stripe__plan_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__refund_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__customer_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
 ```
 
 ### Change the build schema
@@ -139,10 +165,13 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/fivetran_utils
-      version: [">=0.3.0", "<0.4.0"]
+      version: [">=0.4.0", "<0.5.0"]
 
     - package: dbt-labs/dbt_utils
-      version: [">=0.8.0", "<0.9.0"]
+      version: [">=1.0.0", "<2.0.0"]
+
+    - package: dbt-labs/spark_utils
+      version: [">=0.3.0", "<0.4.0"]
 ```
 
 # 🙌 How is this package maintained and can I contribute?

dbt_project.yml

@@ -1,17 +1,14 @@
 config-version: 2
-
 name: 'stripe_source'
-version: '0.7.4'
-
-require-dbt-version: [">=1.0.0", "<2.0.0"]
+version: '0.8.0'
+require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
   stripe_source:
     materialized: table
     +schema: stg_stripe
     tmp:
       materialized: view
-
 vars:
   stripe_source:
     balance_transaction: "{{ source('stripe', 'balance_transaction') }}"
@@ -31,19 +28,10 @@ vars:
     subscription: "{{ source('stripe', 'subscription') }}"
     credit_note: "{{ source('stripe', 'credit_note') }}"
     credit_note_line_item: "{{ source('stripe', 'credit_note_line_item') }}"
-
-
-    #Variables to enable or disable models if you do not have the respective table.
     using_invoices: true
     using_credit_notes: false
     using_payment_method: true
     using_subscriptions: true
-
-    # variable to exclude or include historical subscription data
     stripe__subscription_history: false
-
-    # variable to exclude or include test data (based on `livemode`)
     using_livemode: true
-
-    # variable to disable the filter within invoice line item
     using_invoice_line_sub_filter: true

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'stripe_source_integration_tests'
-version: '0.7.4'
+version: '0.8.0'
 
 
 profile: 'integration_tests'

integration_tests/requirements.txt

@@ -1,7 +1,7 @@
-dbt-snowflake>=1.0.0,<1.3.0
-dbt-bigquery>=1.0.0,<1.3.0
-dbt-redshift>=1.0.0,<1.3.0
-dbt-postgres>=1.0.0,<1.3.0
-dbt-spark>=1.0.0,<1.3.0
-dbt-spark[PyHive]>1.0.0,<1.3.0
-dbt-databricks>=1.0.0,<1.3.0
+dbt-snowflake>=1.3.0,<2.0.0
+dbt-bigquery>=1.3.0,<2.0.0
+dbt-redshift>=1.3.0,<2.0.0
+dbt-postgres>=1.3.0,<2.0.0
+dbt-spark>=1.3.0,<2.0.0
+dbt-spark[PyHive]>=1.3.0,<2.0.0
+dbt-databricks>=1.3.0,<2.0.0

macros/get_balance_transaction_columns.sql

@@ -1,21 +1,21 @@
 {% macro get_balance_transaction_columns() %}
 
 {% set columns = [
-    {"name": "_fivetran_synced", "datatype": dbt_utils.type_timestamp()},
-    {"name": "amount", "datatype": dbt_utils.type_int()},
-    {"name": "available_on", "datatype": dbt_utils.type_timestamp()},
-    {"name": "connected_account_id", "datatype": dbt_utils.type_string()},
-    {"name": "created", "datatype": dbt_utils.type_timestamp()},
-    {"name": "currency", "datatype": dbt_utils.type_string()},
-    {"name": "description", "datatype": dbt_utils.type_string()},
-    {"name": "exchange_rate", "datatype": dbt_utils.type_float()},
-    {"name": "fee", "datatype": dbt_utils.type_int()},
-    {"name": "id", "datatype": dbt_utils.type_string()},
-    {"name": "net", "datatype": dbt_utils.type_int()},
-    {"name": "payout_id", "datatype": dbt_utils.type_string()},
-    {"name": "source", "datatype": dbt_utils.type_string()},
-    {"name": "status", "datatype": dbt_utils.type_string()},
-    {"name": "type", "datatype": dbt_utils.type_string()}
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "amount", "datatype": dbt.type_int()},
+    {"name": "available_on", "datatype": dbt.type_timestamp()},
+    {"name": "connected_account_id", "datatype": dbt.type_string()},
+    {"name": "created", "datatype": dbt.type_timestamp()},
+    {"name": "currency", "datatype": dbt.type_string()},
+    {"name": "description", "datatype": dbt.type_string()},
+    {"name": "exchange_rate", "datatype": dbt.type_float()},
+    {"name": "fee", "datatype": dbt.type_int()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "net", "datatype": dbt.type_int()},
+    {"name": "payout_id", "datatype": dbt.type_string()},
+    {"name": "source", "datatype": dbt.type_string()},
+    {"name": "status", "datatype": dbt.type_string()},
+    {"name": "type", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

macros/get_card_columns.sql

@@ -1,35 +1,35 @@
 {% macro get_card_columns() %}
 
 {% set columns = [
-    {"name": "_fivetran_synced", "datatype": dbt_utils.type_timestamp()},
-    {"name": "account_id", "datatype": dbt_utils.type_string()},
-    {"name": "address_city", "datatype": dbt_utils.type_string()},
-    {"name": "address_country", "datatype": dbt_utils.type_string()},
-    {"name": "address_line_1", "datatype": dbt_utils.type_string()},
-    {"name": "address_line_1_check", "datatype": dbt_utils.type_string()},
-    {"name": "address_line_2", "datatype": dbt_utils.type_string()},
-    {"name": "address_state", "datatype": dbt_utils.type_string()},
-    {"name": "address_zip", "datatype": dbt_utils.type_string()},
-    {"name": "address_zip_check", "datatype": dbt_utils.type_string()},
-    {"name": "brand", "datatype": dbt_utils.type_string()},
-    {"name": "connected_account_id", "datatype": dbt_utils.type_string()},
-    {"name": "country", "datatype": dbt_utils.type_string()},
-    {"name": "created", "datatype": dbt_utils.type_timestamp()},
-    {"name": "currency", "datatype": dbt_utils.type_string()},
-    {"name": "customer_id", "datatype": dbt_utils.type_string()},
-    {"name": "cvc_check", "datatype": dbt_utils.type_string()},
-    {"name": "dynamic_last_4", "datatype": dbt_utils.type_string()},
-    {"name": "exp_month", "datatype": dbt_utils.type_int()},
-    {"name": "exp_year", "datatype": dbt_utils.type_int()},
-    {"name": "fingerprint", "datatype": dbt_utils.type_string()},
-    {"name": "funding", "datatype": dbt_utils.type_string()},
-    {"name": "id", "datatype": dbt_utils.type_string()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "account_id", "datatype": dbt.type_string()},
+    {"name": "address_city", "datatype": dbt.type_string()},
+    {"name": "address_country", "datatype": dbt.type_string()},
+    {"name": "address_line_1", "datatype": dbt.type_string()},
+    {"name": "address_line_1_check", "datatype": dbt.type_string()},
+    {"name": "address_line_2", "datatype": dbt.type_string()},
+    {"name": "address_state", "datatype": dbt.type_string()},
+    {"name": "address_zip", "datatype": dbt.type_string()},
+    {"name": "address_zip_check", "datatype": dbt.type_string()},
+    {"name": "brand", "datatype": dbt.type_string()},
+    {"name": "connected_account_id", "datatype": dbt.type_string()},
+    {"name": "country", "datatype": dbt.type_string()},
+    {"name": "created", "datatype": dbt.type_timestamp()},
+    {"name": "currency", "datatype": dbt.type_string()},
+    {"name": "customer_id", "datatype": dbt.type_string()},
+    {"name": "cvc_check", "datatype": dbt.type_string()},
+    {"name": "dynamic_last_4", "datatype": dbt.type_string()},
+    {"name": "exp_month", "datatype": dbt.type_int()},
+    {"name": "exp_year", "datatype": dbt.type_int()},
+    {"name": "fingerprint", "datatype": dbt.type_string()},
+    {"name": "funding", "datatype": dbt.type_string()},
+    {"name": "id", "datatype": dbt.type_string()},
     {"name": "is_deleted", "datatype": "boolean"},
-    {"name": "last_4", "datatype": dbt_utils.type_string()},
-    {"name": "name", "datatype": dbt_utils.type_string()},
-    {"name": "network", "datatype": dbt_utils.type_string()},
-    {"name": "recipient", "datatype": dbt_utils.type_string()},
-    {"name": "tokenization_method", "datatype": dbt_utils.type_string()}
+    {"name": "last_4", "datatype": dbt.type_string()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "network", "datatype": dbt.type_string()},
+    {"name": "recipient", "datatype": dbt.type_string()},
+    {"name": "tokenization_method", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}