Merge pull request #59 from fivetran/stripe_updates

Stripe updates

.buildkite/scripts/run_models.sh

@@ -19,5 +19,5 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{using_invoice_line_sub_filter: false, using_credit_notes: true}' --full-refresh --target "$db"
+dbt run --vars '{stripe__using_invoice_line_sub_filter: false, stripe__using_credit_notes: true, stripe__using_price: false, stripe__using_subscription_history: false}' --full-refresh --target "$db"
 dbt test --target "$db"

CHANGELOG.md

@@ -1,3 +1,29 @@
+# dbt_stripe_source v0.9.0
+
+[PR #59](https://github.com/fivetran/dbt_stripe_source/pull/59) contains the following changes:
+## 🚨 Breaking Changes 🚨:
+- Variable names have been updated to contain the `stripe` prefix, allowing you to configure global variables while only affecting the Stripe package. 
+
+| **Previous Name**                          | **New Name**                                                                                                                                                                                                                             |
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| using_invoices    | stripe__using_invoices
+| using_credit_notes | stripe__using_credit_notes
+| using_payment_method | stripe__using_payment_method
+| using_livemode | stripe__using_livemode
+| using_invoice_line_sub_filter | stripe__using_invoice_line_sub_filter
+| using_subscriptions | stripe__using_subscriptions
+| using_subscription_history | stripe__using_subscription_history
+
+- `stg_stripe__plan` has been changed to `stg_stripe__price_plan`. Following Stripe's migration from the `Plan` object to the `Price` object ([Stripe doc here.](https://stripe.com/docs/billing/migration/migrating-prices)), we have added a new variable `stripe__using_price` and macro `does_table_exist` that checks if the `price` table exists. This package uses `price` by default if it exists. However, if you still have and wish to keep using `plan`, you can set `stripe__using_price` to False. For more please see the [README](https://github.com/fivetran/dbt_stripe_source#leveraging-plan-vs-price-sources)
+- `stripe__plan_metadata` variable has been renamed to `stripe__price_plan_metadata`
+- Stripe connectors set up after February 09, 2022 will use the `subscription_history` table, as they will no longer be syncing the `subscription` table. This package uses `subscription_history` by default if it exists. However, if you still have the `subscription` table and wish to use it instead, then set the `stripe__using_subscription_history` to False. 
+## 🎉 Feature Updates 🎉: 
+- Added the Union ability to allow for multiple Stripe connectors. The new `source_relation` column in each staging model will specify where each record comes from. For more information please see the [README](https://github.com/fivetran/dbt_stripe_source#unioning-multiple-stripe-connectors) [#33](https://github.com/fivetran/dbt_stripe_source/issues/33)
+- Added new `price` source table in addition to new `stripe__using_price` variable. Stripe migrated the Plan API to Price API (for more information, refer to their [docs](https://stripe.com/docs/billing/migration/migrating-prices)) so we recommend using the price table. The `stripe__using_price` variable and `does_table_exist` macro checks to see if you are indeed using the `price` table. If you plan on using the `plan` table instead you may toggle this to False. For more information on how to configure, refer to the [README](https://github.com/fivetran/dbt_stripe_source#leveraging-plan-vs-price-sources). 
+- Added additional fields to the following models: `stg_stripe__charge`, `stg_stripe__invoice`, `stg_stripe__invoice_line_item`,`stg_stripe__payment_method_card`, `stg_stripe__refund`, `stg_stripe__subscription`.
+
+For more please see the [README](https://github.com/fivetran/dbt_stripe_source/#leveraging-subscription-vs-subscription-history-sources)
+
 # dbt_stripe_source v0.8.0
 
 ## 🚨 Breaking Changes 🚨:

README.md

@@ -22,6 +22,7 @@
 # 🎯 How do I use the dbt package?
 ## Step 1: Prerequisites
 To use this dbt package, you must have the following:
+
 - At least one Fivetran stripe connector syncing data into your destination. 
 - A **BigQuery**, **Snowflake**, **Redshift**, **Databricks**, or **PostgreSQL** destination.
 
@@ -39,7 +40,7 @@ Include the following stripe_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/stripe_source
-    version: [">=0.8.0", "<0.9.0"]
+    version: [">=0.9.0", "<0.10.0"]
 ```
 ## Step 3: Define database and schema variables
 By default, this package runs using your destination and the `stripe` schema. If this is not where your stripe data is (for example, if your stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -50,56 +51,91 @@ vars:
     stripe_schema: your_schema_name 
 ```
 ## Step 4: Disable models for non-existent sources
-This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `subscription`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
+This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `price`, `subscription`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
 
 ```yml
 # dbt_project.yml
 
 ...
 vars:
-    using_invoices:        False  #Disable if you are not using the invoice and invoice_line_item tables
-    using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
-    using_subscriptions:   False  #Disable if you are not using the subscription and plan tables.
-    using_credit_notes:    True   #Enable if you are using the credit note tables.
+    stripe__using_invoices:        False  #Disable if you are not using the invoice and invoice_line_item tables
+    stripe__using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
+    stripe__using_subscriptions:   False  #Disable if you are not using the subscription and plan/price tables.
+    stripe__using_credit_notes:    True   #Enable if you are using the credit note tables.
 
 ```
-## Step 5: Leveraging Subscription Vs Subscription History Sources
-For Stripe connectors set up after February 09, 2022 the `subscription` table has been replaced with the new `subscription_history` table. By default this package will look for your subscription data within the `subscription` source table. However, if you have a newer connector then you must leverage the `stripe__subscription_history` to have the package use the `subscription_history` source rather than the `subscription` table.
-> **Please note that if you have `stripe__subscription_history` enabled then the package will filter for only active records.**
+## (Optional) Step 5: Additional configurations
+<details><summary>Expand to view configurations</summary>
+
+### Unioning Multiple Stripe Connectors
+If you have multiple Stripe connectors you would like to use this package on simultaneously, we have added the ability to do so. Data from disparate connectors will be unioned together and be passed downstream to the end models. The `source_relation` column will specify where each record comes from. To use this functionality, you will need to either set the `stripe_union_schemas` or `stripe_union_databases` variables. Please also make sure the single-source `stripe_database` and `stripe_schema` variables are removed.
+
 ```yml
+# dbt_project.yml
+
+...
+config-version: 2
+
 vars:
-    stripe__subscription_history: True  # False by default. Set to True if your connector syncs the `subscription_history` table. 
+    stripe_union_schemas: ['stripe_us','stripe_mx'] # use this if the data is in different schemas/datasets of the same database/project
+    stripe_union_databases: ['stripe_db_1','stripe_db_2'] # use this if the data is in different databases/projects but uses the same schema name
+```
+### Leveraging Plan vs Price Sources
+
+Customers using Fivetran with the newer [Stripe Price API](https://stripe.com/docs/billing/migration/migrating-prices) will have a `price` table, and possibly a `plan` table if that was used previously. Therefore to accommodate two different source tables we added logic to check if there exists a `price` table by default. If not, it will leverage the `plan` table. However if you wish to use the `plan` table instead, you may set `stripe__using_price` to `false` in your `dbt_project.yml` to override the macro. 
+
+```yml
+# dbt_project.yml
+
+...
+config-version: 2
+
+vars:
+  stripe__using_price: false #  True by default. If true, will look `price ` table. If false, will look for the `plan` table. 
+```
+
+### Leveraging Subscription Vs Subscription History Sources
+For Stripe connectors set up after February 09, 2022 the `subscription` table has been replaced with the new `subscription_history` table. By default this package will look for your subscription data within the `subscription_history` source table. However, if you have an older connector then you must configure the `stripe__using_subscription_history` to `false` in order to have the package use the `subscription` source rather than the `subscription_history` table.
+> **Please note that if you have `stripe__using_subscription_history` enabled then the package will filter for only active records.**
+```yml
+vars:
+    stripe__using_subscription_history: False  # True by default. Set to False if your connector syncs the `subscription` table instead. 
 ```
-## (Optional) Step 6: Additional configurations
-<details><summary>Expand to view configurations</summary>
 
 ### Running on Live vs Test Customers
 By default, this package will run on non-test data (`where livemode = true`) from the source Stripe tables. However, you may want to include and focus on test data when testing out the package or developing your analyses. To run on only test data, add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
     stripe_source:
-        using_livemode: false  # Default = true
+        stripe__using_livemode: false  # Default = true
 ```
 ### Including sub Invoice Line Items
 By default, this package will filter out any records from the `invoice_line_item` source table which include the string `sub_`. This is due to a legacy Stripe issue where `sub_` records were found to be duplicated. However, if you highly utilize these records you may wish they be included in the final output of the `stg_stripe__invoice_line_item` model. To do, so you may include the below variable configuration in your root `dbt_project.yml`:
 ```yml
 vars:
     stripe_source:
-        using_invoice_line_sub_filter: false # Default = true
+        stripe__using_invoice_line_sub_filter: false # Default = true
 ```
 
 ### Pivoting out Metadata Properties
 Oftentimes you may have custom fields within your source tables that is stored as a JSON object that you wish to pass through. By leveraging the `metadata` variable, this package pivot out fields into their own columns. The metadata variables accept dictionaries in addition to strings.
 
-Additionally, if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field, Below are examples of how you would add the respective fields.
+Additionally, you may `alias` your field if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field. Below are examples of how you would add the respective fields.
 
-The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
+The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `price`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
 
 ```yml
 vars: 
+  stripe__account_metadata:
+    - name: metadata_field
+    - name: another_metadata_field
+    - name: and_another_metadata_field
   stripe__charge_metadata:
     - name: metadata_field_1
+  stripe__customer_metadata:
+    - name: metadata_field_6
+      alias: metadata_field_six
   stripe__invoice_metadata: 
     - name: metadata_field_2
   stripe__payment_intent_metadata:
@@ -111,25 +147,22 @@ vars:
   stripe__payout_metadata:
     - name: 123
       alias: one_two_three
-  stripe__plan_metadata:
-    - name: rename
-    - alias: renamed_field
+  stripe__price_plan_metadata: ## Used for both Price and Plan sources
+    - name: rename_price
+      alias: renamed_field_price
   stripe__refund_metadata:
     - name: metadata_field_3
-    - name: metadata_field_4
   stripe__subscription_metadata:
-    - name: metadata_field_5
-  stripe__customer_metadata:
-    - name: metadata_field_6
+    - name: 567
+      alias: five_six_seven
 
 ```
 
 Alternatively, if you only have strings in your JSON object, the metadata variable accepts the following configuration as well.
->**Note**: `stripe__plan_metadata` is only shown below, but the format will work for all metadata variables. 
 
 ```yml
 vars:
-    stripe__plan_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
+    stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
 ```
 
 ### Change the build schema
@@ -152,7 +185,7 @@ vars:
 
 </details>
 
-## (Optional) Step 7: Orchestrate your models with Fivetran Transformations for dbt Core™
+## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand to view details</summary>
 <br>
 

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'stripe_source'
-version: '0.8.0'
+version: '0.9.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
@@ -11,9 +11,12 @@ models:
       materialized: view
 vars:
   stripe_source:
+    account: "{{ source('stripe', 'account') }}"
     balance_transaction: "{{ source('stripe', 'balance_transaction') }}"
     card: "{{ source('stripe', 'card') }}"
     charge: "{{ source('stripe', 'charge') }}"
+    credit_note: "{{ source('stripe', 'credit_note') }}"
+    credit_note_line_item: "{{ source('stripe', 'credit_note_line_item') }}"
     customer: "{{ source('stripe', 'customer') }}"
     fee: "{{ source('stripe', 'fee') }}"
     invoice: "{{ source('stripe', 'invoice') }}"
@@ -23,15 +26,7 @@ vars:
     payment_method: "{{ source('stripe', 'payment_method') }}"
     payout: "{{ source('stripe', 'payout') }}"
     plan: "{{ source('stripe', 'plan') }}"
+    price: "{{ source('stripe', 'price') }}"
     refund: "{{ source('stripe', 'refund') }}"
     subscription_history: "{{ source('stripe', 'subscription_history') }}"
-    subscription: "{{ source('stripe', 'subscription') }}"
-    credit_note: "{{ source('stripe', 'credit_note') }}"
-    credit_note_line_item: "{{ source('stripe', 'credit_note_line_item') }}"
-    using_invoices: true
-    using_credit_notes: false
-    using_payment_method: true
-    using_subscriptions: true
-    stripe__subscription_history: false
-    using_livemode: true
-    using_invoice_line_sub_filter: true
+    subscription: "{{ source('stripe', 'subscription') }}"

integration_tests/dbt_project.yml

@@ -1,14 +1,15 @@
 config-version: 2
 
 name: 'stripe_source_integration_tests'
-version: '0.8.0'
+version: '0.9.0'
 
 
 profile: 'integration_tests'
 
 vars:
   stripe_schema: stripe_source_integrations_tests
   stripe_source:
+    stripe_account_identifier:                "account_data"
     stripe_group_identifier:                  "group_data"
     stripe_balance_transaction_identifier:    "balance_transaction_data"
     stripe_card_identifier:                   "card_data"
@@ -22,6 +23,7 @@ vars:
     stripe_payment_method_identifier:         "payment_method_data"
     stripe_payout_identifier:                 "payout_data"
     stripe_plan_identifier:                   "plan_data"
+    stripe_price_identifier:                  "price_data"
     stripe_refund_identifier:                 "refund_data"
     stripe_subscription_history_identifier:   "subscription_history_data"
     stripe_subscription_identifier:           "subscription_data"
@@ -33,6 +35,16 @@ seeds:
   stripe_source_integration_tests:
     +column_types:
       _fivetran_synced: timestamp
+      created: timestamp
+      voided_at: timestamp
+    account_data:
+      +column_types:
+        id: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+        company_address_postal_code: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+        individual_id: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+    invoice_data:
+      +column_types:
+        id: "{{ 'varchar(100)' if target.name in ('redshift', 'postgres') else 'string' }}"
 
 dispatch:
   - macro_namespace: dbt_utils