Stripe updates

.buildkite/scripts/run_models.sh

@@ -19,5 +19,5 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{using_invoice_line_sub_filter: false, using_credit_notes: true}' --full-refresh --target "$db"
+dbt run --vars '{stripe__using_invoice_line_sub_filter: false, stripe__using_credit_notes: true, stripe__price: False, stripe__using_subscription_history: false}' --full-refresh --target "$db"
 dbt test --target "$db"

CHANGELOG.md

@@ -1,3 +1,28 @@
+# dbt_stripe_source v0.9.0
+
+[PR #59](https://github.com/fivetran/dbt_stripe_source/pull/59) contains the following changes:
+## 🎉 Feature Updates 🎉: 
+- Added the Union ability to allow for multiple Stripe connectors. The new `source_relation` column in each staging model will specify where each record comes from. For more information please see the [README](https://github.com/fivetran/dbt_stripe_source#unioning-multiple-stripe-connectors) [#33](https://github.com/fivetran/dbt_stripe_source/issues/33)
+- Added new `price` source table
+- Added additional fields to the following models: `stg_stripe__charge`, `stg_stripe__invoice` [#53](https://github.com/fivetran/dbt_stripe_source/issues/53), `stg_stripe__invoice_line_item`,`stg_stripe__payment_method_card`, `stg_stripe__refund`, `stg_stripe__subscription`.
+
+## 🚨 Breaking Changes 🚨:
+- Variable names have been updated to contain the `stripe` prefix, allowing you to configure global variables while only affecting the Stripe package. 
+
+| **Previous Name**                          | **New Name**                                                                                                                                                                                                                             |
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| using_invoices    | stripe__using_invoices
+| using_credit_notes | stripe__using_credit_notes
+| using_payment_method | stripe__using_payment_method
+| using_livemode | stripe__using_livemode
+| using_invoice_line_sub_filter | stripe__using_invoice_line_sub_filter
+| using_subscriptions | stripe__using_subscriptions
+| using_subscription_history | stripe__using_subscription_history
+| using_price | stripe__using_price
+
+- `stg_stripe__plan` has been changed to `stg_stripe__pricing`. Following Stripe's migration from the `Plan` object to the `Price` object ([Stripe doc here.](https://stripe.com/docs/billing/migration/migrating-prices)), we have added a new variable `stripe__price` and macro `does_table_exist` that checks if the `price` table exists. This package uses `price` by default if exists. However, if you still have and wish to keep using `plan` which we recommend, you can set `stripe__price` to False. For more please see the [README](https://github.com/fivetran/dbt_stripe_source#leveraging-plan-vs-price-sources)
+- Stripe connectors set up after February 09, 2022 will use the `subscription_history` table, as they will no longer be syncing the `subscription` table. This package uses `subscription_history` by default if exists. However, if you still have the `subscription` table and wish to use it instead, then set the `stripe__using_subscription_history` to False. For more please see the [README](https://github.com/fivetran/dbt_stripe_source/#leveraging-subscription-vs-subscription-history-sources)
+
 # dbt_stripe_source v0.8.0
 
 ## 🚨 Breaking Changes 🚨:

README.md

@@ -39,7 +39,7 @@ Include the following stripe_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/stripe_source
-    version: [">=0.8.0", "<0.9.0"]
+    version: [">=0.9.0", "<0.10.0"]
 ```
 ## Step 3: Define database and schema variables
 By default, this package runs using your destination and the `stripe` schema. If this is not where your stripe data is (for example, if your stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -50,56 +50,95 @@ vars:
     stripe_schema: your_schema_name 
 ```
 ## Step 4: Disable models for non-existent sources
-This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `subscription`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
+This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `price`, `subscription`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
 
 ```yml
 # dbt_project.yml
 
 ...
 vars:
-    using_invoices:        False  #Disable if you are not using the invoice and invoice_line_item tables
-    using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
-    using_subscriptions:   False  #Disable if you are not using the subscription and plan tables.
-    using_credit_notes:    True   #Enable if you are using the credit note tables.
+    stripe__using_invoices:        False  #Disable if you are not using the invoice and invoice_line_item tables
+    stripe__using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
+    stripe__using_subscriptions:   False  #Disable if you are not using the subscription and plan/pricing tables.
+    stripe__using_credit_notes:    True   #Enable if you are using the credit note tables.
 
 ```
-## Step 5: Leveraging Subscription Vs Subscription History Sources
-For Stripe connectors set up after February 09, 2022 the `subscription` table has been replaced with the new `subscription_history` table. By default this package will look for your subscription data within the `subscription` source table. However, if you have a newer connector then you must leverage the `stripe__subscription_history` to have the package use the `subscription_history` source rather than the `subscription` table.
-> **Please note that if you have `stripe__subscription_history` enabled then the package will filter for only active records.**
+## (Optional) Step 5: Additional configurations
+<details><summary>Expand to view configurations</summary>
+
+### Leveraging Plan vs Price Sources
+
+Customers using Fivetran with the newer Stripe Price API will have a `price` table in place of the older `plan` table. Therefore to accommodate two different source tables we added additional logic in the `stg_stripe__pricing` model, which replaces the `stg_stripe__plan` model. This model checks if there exists a `price` table using a new `does_table_exist()` macro. If not, it will look for a `plan` table. The default is to use the `price` table if it exists. However if you wish to use the `plan` table instead, you may set `stripe__using_price` to `false` in your `dbt_project.yml` to override the macro. 
+
+We recommend using the `price` table as Stripe replaced the Plans API with the Price API and is backwards compatible.
+
+```yml
+# dbt_project.yml
+
+...
+config-version: 2
+
+vars:
+  stripe:
+    stripe__using_price: false #  True by default. If true, will look `price ` table. If false, will look for the `plan` table. 
+```
+
+### Leveraging Subscription Vs Subscription History Sources
+For Stripe connectors set up after February 09, 2022 the `subscription` table has been replaced with the new `subscription_history` table. By default this package will look for your subscription data within the `subscription_history` source table. However, if you have a older connector then you must leverage the `stripe__subscription` to have the package use the `subscription` source rather than the `subscription_history` table.
+> **Please note that if you have `stripe__using_subscription_history` enabled then the package will filter for only active records.**
 ```yml
 vars:
-    stripe__subscription_history: True  # False by default. Set to True if your connector syncs the `subscription_history` table. 
+    stripe__using_subscription_history: False  # True by default. Set to False if your connector syncs the `subscription` table instead. 
+```
+
+### Unioning Multiple Stripe Connectors
+If you have multiple Stripe connectors you would like to use this package on simultaneously, we have added the ability to do so. Data from disparate connectors will be unioned together and be passed downstream to the end models. The `source_relation` column will specify where each record comes from. To use this functionality, you will need to either set the `stripe_union_schemas` or `stripe_union_databases` variables. Please also make sure the single-source `stripe_database` and `stripe_schema` variables are removed.
+
+```yml
+# dbt_project.yml
+
+...
+config-version: 2
+
+vars:
+    stripe_union_schemas: ['stripe_us','stripe_mx'] # use this if the data is in different schemas/datasets of the same database/project
+    stripe_union_databases: ['stripe_db_1','stripe_db_2'] # use this if the data is in different databases/projects but uses the same schema name
 ```
-## (Optional) Step 6: Additional configurations
-<details><summary>Expand to view configurations</summary>
 
 ### Running on Live vs Test Customers
 By default, this package will run on non-test data (`where livemode = true`) from the source Stripe tables. However, you may want to include and focus on test data when testing out the package or developing your analyses. To run on only test data, add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
     stripe_source:
-        using_livemode: false  # Default = true
+        stripe__using_livemode: false  # Default = true
 ```
 ### Including sub Invoice Line Items
 By default, this package will filter out any records from the `invoice_line_item` source table which include the string `sub_`. This is due to a legacy Stripe issue where `sub_` records were found to be duplicated. However, if you highly utilize these records you may wish they be included in the final output of the `stg_stripe__invoice_line_item` model. To do, so you may include the below variable configuration in your root `dbt_project.yml`:
 ```yml
 vars:
     stripe_source:
-        using_invoice_line_sub_filter: false # Default = true
+        stripe__using_invoice_line_sub_filter: false # Default = true
 ```
 
 ### Pivoting out Metadata Properties
 Oftentimes you may have custom fields within your source tables that is stored as a JSON object that you wish to pass through. By leveraging the `metadata` variable, this package pivot out fields into their own columns. The metadata variables accept dictionaries in addition to strings.
 
-Additionally, if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field, Below are examples of how you would add the respective fields.
+Additionally, you may `alias` your field if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field. Below are examples of how you would add the respective fields.
 
-The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
+The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `price`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
 
 ```yml
 vars: 
+  stripe__account_metadata:
+    - name: metadata_field
+    - name: another_metadata_field
+    - name: and_another_metadata_field
   stripe__charge_metadata:
     - name: metadata_field_1
+  stripe__customer_metadata:
+    - name: metadata_field_6
+      alias: metadata_field_six
   stripe__invoice_metadata: 
     - name: metadata_field_2
   stripe__payment_intent_metadata:
@@ -113,14 +152,15 @@ vars:
       alias: one_two_three
   stripe__plan_metadata:
     - name: rename
-    - alias: renamed_field
+      alias: renamed_field
+  stripe__price_metadata:
+    - name: rename_price
+      alias: renamed_field_price
   stripe__refund_metadata:
     - name: metadata_field_3
-    - name: metadata_field_4
   stripe__subscription_metadata:
-    - name: metadata_field_5
-  stripe__customer_metadata:
-    - name: metadata_field_6
+    - name: 567
+      alias: five_six_seven
 
 ```
 
@@ -151,7 +191,7 @@ vars:
 
 </details>
 
-## (Optional) Step 7: Orchestrate your models with Fivetran Transformations for dbt Core™
+## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand to view details</summary>
 <br>
 

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'stripe_source'
-version: '0.8.0'
+version: '0.9.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
@@ -11,9 +11,12 @@ models:
       materialized: view
 vars:
   stripe_source:
+    account: "{{ source('stripe', 'account') }}"
     balance_transaction: "{{ source('stripe', 'balance_transaction') }}"
     card: "{{ source('stripe', 'card') }}"
     charge: "{{ source('stripe', 'charge') }}"
+    credit_note: "{{ source('stripe', 'credit_note') }}"
+    credit_note_line_item: "{{ source('stripe', 'credit_note_line_item') }}"
     customer: "{{ source('stripe', 'customer') }}"
     fee: "{{ source('stripe', 'fee') }}"
     invoice: "{{ source('stripe', 'invoice') }}"
@@ -23,15 +26,7 @@ vars:
     payment_method: "{{ source('stripe', 'payment_method') }}"
     payout: "{{ source('stripe', 'payout') }}"
     plan: "{{ source('stripe', 'plan') }}"
+    price: "{{ source('stripe', 'price') }}"
     refund: "{{ source('stripe', 'refund') }}"
     subscription_history: "{{ source('stripe', 'subscription_history') }}"
-    subscription: "{{ source('stripe', 'subscription') }}"
-    credit_note: "{{ source('stripe', 'credit_note') }}"
-    credit_note_line_item: "{{ source('stripe', 'credit_note_line_item') }}"
-    using_invoices: true
-    using_credit_notes: false
-    using_payment_method: true
-    using_subscriptions: true
-    stripe__subscription_history: false
-    using_livemode: true
-    using_invoice_line_sub_filter: true
+    subscription: "{{ source('stripe', 'subscription') }}"

integration_tests/dbt_project.yml

@@ -1,14 +1,15 @@
 config-version: 2
 
 name: 'stripe_source_integration_tests'
-version: '0.8.0'
+version: '0.9.0'
 
 
 profile: 'integration_tests'
 
 vars:
   stripe_schema: stripe_source_integrations_tests
   stripe_source:
+    stripe_account_identifier:                "account_data"
     stripe_group_identifier:                  "group_data"
     stripe_balance_transaction_identifier:    "balance_transaction_data"
     stripe_card_identifier:                   "card_data"
@@ -22,6 +23,7 @@ vars:
     stripe_payment_method_identifier:         "payment_method_data"
     stripe_payout_identifier:                 "payout_data"
     stripe_plan_identifier:                   "plan_data"
+    stripe_price_identifier:                  "price_data"
     stripe_refund_identifier:                 "refund_data"
     stripe_subscription_history_identifier:   "subscription_history_data"
     stripe_subscription_identifier:           "subscription_data"
@@ -33,6 +35,16 @@ seeds:
   stripe_source_integration_tests:
     +column_types:
       _fivetran_synced: timestamp
+      created: timestamp
+      voided_at: timestamp
+    account_data:
+      +column_types:
+        id: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+        company_address_postal_code: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+        individual_id: "{%- if target.type == 'bigquery' -%} INT64 {%- else -%} bigint {%- endif -%}"
+    invoice_data:
+      +column_types:
+        id: "{{ 'varchar(100)' if target.name in ('redshift', 'postgres') else 'string' }}"
 
 dispatch:
   - macro_namespace: dbt_utils