Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(dbt): Keep syncing datasets if one failed to sync columns #313

Merged
merged 1 commit into from
Aug 19, 2024

Conversation

Vitor-Avila
Copy link
Contributor

When syncing datasets with the --preserve-metadata or --merge-metadata flags, a request to sync the dataset columns is triggered. Currently, in case this request fails the command execution is aborted.

This PR gracefully handles failures, adding the failed dataset to the list of failed_datasets (which can be listed at the end in case --raise-failures is used).

refreshed_columns_list,
)
except SupersetError:
failed_datasets.append(model["unique_id"])
Copy link

@Antonio-RiveroMartnez Antonio-RiveroMartnez Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not very familiar with this part of the code, but since this list is visible only using a certain flag (--raise-failures) and it wasn't there before to consume, it would be good to put that info somewhere in the help of the cli method if not there already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the feedback, @Antonio-RiveroMartnez 🙏 I can confirm the flag is documented in the help command:

(local-cli) ➜  backend-sdk git:(dbt-cloud-versionless) preset-cli superset sync dbt-core --help
Usage: preset-cli superset sync dbt-core [OPTIONS] FILE

Options:
  --project TEXT              Name of the dbt project
  --target TEXT               Target name
  --profiles PATH             Location of profiles.yml file
  --exposures PATH            Path to file where exposures will be written
  --import-db                 Import (or update) the database connection to
                              Superset
  --disallow-edits            Mark resources as managed externally to prevent
                              edits
  --external-url-prefix TEXT  Base URL for resources
  -s, --select TEXT           Model selection
  -x, --exclude TEXT          Models to exclude
  --exposures-only            Do not sync models to datasets and only fetch
                              exposures instead
  --preserve-columns          Preserve column and metric configurations
                              defined in Preset
  --preserve-metadata         Preserve column and metric configurations
                              defined in Preset
  --merge-metadata            Update Preset configurations based on dbt
                              metadata. Preset-only metrics are preserved
  --raise-failures            End the execution with an error if a model fails
                              to sync or a deprecated feature is used
  --help                      Show this message and exit.

Since this is a customer-facing CLI, we also have this logic documented here.

@Vitor-Avila Vitor-Avila merged commit efc3253 into main Aug 19, 2024
5 checks passed
@Vitor-Avila Vitor-Avila deleted the continue-sync branch August 19, 2024 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants