Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dbt): Support defining default values for columns and metrics #325

Merged
merged 3 commits into from
Nov 7, 2024

Conversation

Vitor-Avila
Copy link
Contributor

@Vitor-Avila Vitor-Avila commented Nov 6, 2024

This PR adds a new feature for the dbt sync: the ability to specify default configs for metrics and columns. This allows you to have default values at the model level, and then if needed overwrite these for specific columns and metrics.

To re-cap, there are 3 dbt sync behaviors:

Default (reload_columns is set to True and merge_metadata is set to False)

All metrics defined in dbt are synced to Superset (if the metric already exists in Superset, it gets entirely replaced by dbt's metadata). Column definitions from dbt are also synced to Superset.

In this flow, metric default configs would be applied to all metrics (unless the config is defined at the metric level), and column default configs would be applied to all columns (unless the config is defined at the column level) - even columns that are not defined in dbt.

Merge metadata (reload_columns is set to False and merge_metadata is set to True)

All metrics defined in dbt are synced to Superset (if the metric already exists in Superset, it gets entirely replaced by dbt's metadata). Column definitions from dbt are also synced to Superset.

In this flow, metric default configs would be applied to all metrics (even the ones that only exist in Superset), and column default configs would be applied to all columns (even columns not defined in dbt).

Preserve metadata (both reload_columns and merge_metadataare set toFalse`)

In this flow, Superset is considered the source of truth. Metrics are only synced from dbt if they don't exist in Superset yet. Column definitions are only synced from dbt if not defined in Superset yet.

Here, metric default configs are only applied to dbt metrics that are getting synced (new metrics), and to columns that are defined in dbt and getting synced.

Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@Vitor-Avila Vitor-Avila merged commit 4827024 into main Nov 7, 2024
5 checks passed
@Vitor-Avila Vitor-Avila deleted the feat/default-configs branch November 7, 2024 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants