Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug/databricks syntax #33

Merged
merged 14 commits into from
Apr 1, 2024
Merged

Conversation

fivetran-catfritz
Copy link
Contributor

@fivetran-catfritz fivetran-catfritz commented Mar 15, 2024

PR Overview

This PR will address the following Issue/Feature:
#31

This PR will result in the following new package version:

  • Following batch release versioning.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Bug Fix

  • Removed the use of ignore nulls statements in marketo__lead_history and marketo__change_data_scd, which was incompatible with PostgreSQL and Databricks Runtime. The logic has been updated with a new approach but produces the same results as before.
  • Updated model marketo__change_data_pivot to use the activity_id as a tie-breaker to remove randomness when ordering events having the same activity_timestamp.
    • Previously if two events happened at the same timestamp, results would be inconsistent, which propagated to downstream models. Now, this model will produce consistent results.

Under the hood

  • Added additional variable configurations to integration tests to account for a wider range of situations.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

  • See internal ticket

If you had to summarize this PR in an emoji, which would it be?

💃

@fivetran-catfritz fivetran-catfritz self-assigned this Mar 15, 2024
Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-catfritz great job working through this PR! I do still feel like there is some more around the Marketo data model we need to understand better. Would you be able to open a FR for us to explore the data model in more detail and determine if we need to rework any of the logic.

Regardless, these changes look good to me! Let's merge these into the release branch and ensure they will work with the next release changes. If they do, we can move forward with starting the release process.

Comment on lines +2 to +6
# version: [">=0.11.0", "<0.12.0"]
- git: https://github.com/fivetran/dbt_marketo_source.git
revision: update/customer360
warn-unpinned: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to swap before release

@fivetran-catfritz fivetran-catfritz merged commit c2782f0 into release/v0.11.0 Apr 1, 2024
8 checks passed
fivetran-catfritz added a commit that referenced this pull request Apr 2, 2024
* document default LEAD columns (#32)

* changes

* docs

* changelog

* update source package ref

* Update CHANGELOG.md

Co-authored-by: fivetran-catfritz <[email protected]>

* changelog

---------

Co-authored-by: fivetran-catfritz <[email protected]>

* Bug/databricks syntax (#33)

* changes

* docs

* changelog

* update source package ref

* Update CHANGELOG.md

Co-authored-by: fivetran-catfritz <[email protected]>

* changelog

* bug/databricks-syntax

* update pkg

* update cte

* remove ignore nulls

* updates

* updates & regen docs

* update yml

---------

Co-authored-by: Jamie Rodriguez <[email protected]>

* Update packages.yml

---------

Co-authored-by: Jamie Rodriguez <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants