[Feature] Add non-standard columns to Card

[bramrodenburg]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The columns description, iin and issuer are by default not available in the Card table. Users first need to contact Stripe, after which Fivetran also picks up these columns. It would be great if there are flags that allow users of the package to add these columns to the stg_stripe__card model.

How would you implement this feature?

Add boolean variables that allow users to add description, iin and issuer to the stg_stripe__card model. Note that these will need to be separate variables, since these columns can be enabled by Stripe on an individual basis.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance.
• No.

Anything else?

See the Fivetran documentation here: https://fivetran.com/docs/connectors/applications/stripe

[fivetran-avinash]

Hi @bramrodenburg ! Thanks for opening this issue and contributing a potential PR here. I'd be a little wary about adding too many variables to this model as it becomes a bit complex for customers to manage.

I think we can take a few approaches here that are more sustainable.

1. We bring the fields into our get_card_columns macro and stg_stripe__card model. The macro ensures that even if these fields do not exist, they will return null values.

However, since these are non-standard fields and will not be used the wide majority of our customers, I don't feel as comfortable bringing these in by default.

2. The preferred approach I'd have is to add passthrough column functionality for these above models so that you can define these fields as additional columns you'd want to bring into stg_stripe__card in your dbt_project.yml. We haven't utilized passthrough columns yet in Stripe, but you can see them in action in Shopify (documentation here, model example here). But that way only customers that utilize these fields will be the only ones to see them.

So my preferred approach would be (2). Let me know if that makes sense or if you have any questions! But I think we are definitely close to scoping this task out and bringing it into a coming sprint.

[bramrodenburg]

The second approach would have my preference over the approach I originally suggested. Would this be something I could give a go on my open pull request, or is this something you would pick up in a coming sprint? I am more than happy to update my PR and contribute.

[fivetran-avinash]

Hi @bramrodenburg ! If you are able to have time to update your PR and add passthrough functionality, that'd be great!

You can use the template from our other packages to update this PR. Here are some guidance steps:

1. You will want to add the fivetran_utils.add_pass_through_columns macro (example in L28 of this Shopify macro) to the get_card_columns macro here.

I'd also recommend naming the variable card_pass_through_columns.

For more context, this macro creates the proper name, datatype, and aliasing for user defined pass through column variable. This macro allows for pass through variables to be more dynamic and allow users to alias custom fields they are bringing in.

2. You will want to add the fivetran_utils.fill_pass_through_columns macro (example [in L59 of this Shopify macro] to the stg_shopify__card model [in this space here].

For more context, this macro is used to generate the correct sql for package staging models for user defined pass through columns.

3. Add the Passing through Additional Fields README section in Section 5 of the README. You can basically copy the shopify text section, then modify the code block to only reference the new card_pass_through_columns section.

4. Bump up the dbt_project.yml versions up to 0.12.1.

5. Update the CHANGELOG documenting these changes. You can use this as a reference. https://github.com/fivetran/dbt_shopify_source/blame/main/CHANGELOG.md#L126-L134

Hope that's helpful! It shouldn't take too long despite all these steps, but let me know if you have any questions.

[bramrodenburg]

Thanks a lot for the detailed steps. It made it super straightforward to implement it. Incorporated it into my PR.

[fivetran-avinash]

Thanks @bramrodenburg ! Glad it was straightforward and you were able to verify that everything works as planned.

I think the plan is we will bring this PR into our review for a coming sprint, as we have multiple Stripe issues we want to address concurrently (including I think another one of yours!), so we will batch them together so they can all go live at once. The changes should be straightforward to verify so hopefully it shouldn't take long for us to get this ready to release.