fix: remove deprecated overwrite_schema configuration which has incorrect behavior #2554
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rect behavior Uses of mode='append' and overwrite_schema=True lead to inconsistent behavior between Rust and PyArrow engines for write_deltalake. In the PyArrow case the parameter is quietly omitted so users may experience unexpected behavior since schemas will not actually be overridden. Users of this parameter set most likely want schema_mode='merge' which would allow for schema evolution on appends to a Delta Table Fixes delta-io#2553
github-actions
bot
added
binding/python
…elease Many of the subcrates are unafected by any changes here, so I'm just expanding their compatibility range.
rtyler
requested review from
wjones127,
fvaleye,
roeap and
ion-elgreco
as code owners
ion-elgreco
approved these changes
May 29, 2024
github-merge-queue bot
pushed a commit
to Unstructured-IO/unstructured
that referenced
this pull request
Jun 13, 2024
### Summary Closes #3173. Removes the `overwrite_schema` kwarg from the Delta Table connector and bumps the `deltalake` version. Per [this PR](delta-io/delta-rs#2554) in the `deltalake` repo, the `overwrite_schema` kwarg is deprecated as of version `0.18.0`. Users can specify `schema_mode="merge"` to obtain the same behavior. - `schema_mode="merge"` is equivalent to `overwrite_schema=False` - `schema_mode="overwrite"` is equivalent to `overwrite_schema=True` Also adds an `engine` parameter that you can use to set `"rust"` or `"pyarrow"` as the engine. `engine` defaults to `"pyarrow"` and `schema_mode` defaults to `None`, which is consistent with the behavior in `deltalake` documented [here](https://delta-io.github.io/delta-rs/api/delta_writer/). ### Testing The Delta Table ingest tests should pass on this PR. --------- Co-authored-by: Ahmet Melek <[email protected]>
github-merge-queue bot
pushed a commit
to Unstructured-IO/unstructured
that referenced
this pull request
Jun 13, 2024
### Summary Closes #3173. Removes the `overwrite_schema` kwarg from the Delta Table connector and bumps the `deltalake` version. Per [this PR](delta-io/delta-rs#2554) in the `deltalake` repo, the `overwrite_schema` kwarg is deprecated as of version `0.18.0`. Users can specify `schema_mode="merge"` to obtain the same behavior. - `schema_mode="merge"` is equivalent to `overwrite_schema=False` - `schema_mode="overwrite"` is equivalent to `overwrite_schema=True` Also adds an `engine` parameter that you can use to set `"rust"` or `"pyarrow"` as the engine. `engine` defaults to `"pyarrow"` and `schema_mode` defaults to `None`, which is consistent with the behavior in `deltalake` documented [here](https://delta-io.github.io/delta-rs/api/delta_writer/). ### Testing The Delta Table ingest tests should pass on this PR. --------- Co-authored-by: Ahmet Melek <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses of mode='append' and overwrite_schema=True lead to inconsistent behavior between Rust and PyArrow engines for write_deltalake. In the PyArrow case the parameter is quietly omitted so users may experience unexpected behavior since schemas will not actually be overridden.
Users of this parameter set most likely want schema_mode='merge' which would allow for schema evolution on appends to a Delta Table
Fixes #2553