fix(ingest/unity): Remove metastore from ingestion and urns; standardize platform instance; add notebook filter #8943
Conversation
github-actions
bot
added
ingestion
…ize platform instance; add notebook filter
asikowitz
force-pushed
the
databricks-remove-metastore
branch
from
6116c18
to
d224996
Compare
docs/how/updating-datahub.md
Outdated
|
|
||
| ### Breaking Changes | ||
|
|
||
| - #8810 - Removed support for SQLAlchemy 1.3.x. Only SQLAlchemy 1.4.x is supported now. | ||
| - #8853 - The Airflow plugin no longer supports Airflow 2.0.x or Python 3.7. See the docs for more details. | ||
| - #8853 - Introduced the Airflow plugin v2. If you're using Airflow 2.3+, the v2 plugin will be enabled by default, and so you'll need to switch your requirements to include `pip install 'acryl-datahub-airflow-plugin[plugin-v2]'`. To continue using the v1 plugin, set the `DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN` environment variable to `true`. | ||
| - #8943 All Unity Catalog urns are changed if a new config param, `include_metastore`, is set to `false`. | ||
| This is set to `true` by default at the moment, but this default will be changed in the future. | ||
| To handle the change in urns, we recommend soft deleting all databricks data via the DataHub CLI: `datahub delete --platform databricks --soft`, |
There was a problem hiding this comment.
add a note that if you have stateful ingest enabled, you don't need to do anything
| " All databricks urns will change if you re-ingest with this disabled." | ||
| " We recommend soft deleting all databricks data and re-ingesting with include_metastore set to False." | ||
| ) | ||
| logger.warning(msg) |
There was a problem hiding this comment.
let's also call add_global_warning
| @@ -247,10 +251,14 @@ def build_service_principal_map(self) -> None: | |||
|
|
|||
| def process_notebooks(self) -> Iterable[MetadataWorkUnit]: | |||
| for notebook in self.unity_catalog_api_proxy.workspace_notebooks(): | |||
| if not self.config.notebook_pattern.allowed(notebook.path): | |||
There was a problem hiding this comment.
was this just missing before?
There was a problem hiding this comment.
I added the pattern as part of this PR, forgot it last time
docs/how/updating-datahub.md
Outdated
|
|
||
| ### Breaking Changes | ||
|
|
||
| - #8810 - Removed support for SQLAlchemy 1.3.x. Only SQLAlchemy 1.4.x is supported now. | ||
| - #8853 - The Airflow plugin no longer supports Airflow 2.0.x or Python 3.7. See the docs for more details. | ||
| - #8853 - Introduced the Airflow plugin v2. If you're using Airflow 2.3+, the v2 plugin will be enabled by default, and so you'll need to switch your requirements to include `pip install 'acryl-datahub-airflow-plugin[plugin-v2]'`. To continue using the v1 plugin, set the `DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN` environment variable to `true`. | ||
| - #8943 All Unity Catalog urns are changed if a new config param, `include_metastore`, is set to `false`. |
There was a problem hiding this comment.
The phrasing is a bit confusing, and splitting this across the two sections isn't ideal
maybe let's do "The unity catalog ingestion source has a new option include_metastore, which will cause all urns to be changed when disabled. This is currently enabled by default to preserve compatibility, but will be disabled by default and then removed in the future. If stateful ingestion is enabled, simply setting include_metastore: true will perform all required cleanup. Otherwise, we recommend soft deleting all databricks data via the DataHub CLI: datahub delete --platform databricks --soft and then reingest with include_metastore: false.
asikowitz
added
the
merge-pending-ci
maggiehays
added
the
hacktoberfest-accepted
It's not pretty but it gets the job done...
Checklist