Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Schema evolution not coercing with Large arrow types #2298

Closed
ion-elgreco opened this issue Mar 17, 2024 · 2 comments · Fixed by #2305
Closed

Schema evolution not coercing with Large arrow types #2298

ion-elgreco opened this issue Mar 17, 2024 · 2 comments · Fixed by #2305
Labels
bug Something isn't working

Comments

@ion-elgreco
Copy link
Collaborator

ion-elgreco commented Mar 17, 2024 •
edited
Loading

Environment

Delta-rs version:

Binding: 0.16.0

Bug

What happened:
Writing a delta table that has a partition column which is primitive type String and then trying to write with schema_mode="merge" to add new columns results in, when your partition column is in arrow dtype LargeUtf8

SchemaMismatchError: Schema error: Fail to merge schema field 'bar' because the from data_type = LargeUtf8 does not equal Dictionary(UInt16, Utf8)
Schema error: Fail to merge schema field 'bar' because the from data_type = LargeUtf8 does not equal Dictionary(UInt16, Utf8)
@ion-elgreco ion-elgreco added the bug Something isn't working label Mar 17, 2024
@ion-elgreco ion-elgreco changed the title Schema evolution not coercing with Large types Schema evolution not coercing with Large arrow types Mar 17, 2024
@ion-elgreco
Copy link
Collaborator Author

@aersam fyi :)

@aersam
Copy link
Contributor

aersam commented Mar 20, 2024

My boss hit this one and I introduced it - I better fix it ;)

aersam added a commit to aersam/delta-rs that referenced this issue Mar 20, 2024
@aersam
Fixes delta-io#2298
55c8450
@aersam aersam mentioned this issue Mar 20, 2024
Merged
ion-elgreco pushed a commit to aersam/delta-rs that referenced this issue Mar 20, 2024
@aersam @ion-elgreco
Fixes delta-io#2298
d4811c3
ion-elgreco pushed a commit that referenced this issue Mar 21, 2024
@aersam
fix: schema evolution not coercing with large arrow types (#2305)
6a51971 
# Description
This fixes schema merge for large arrow types in combination with
dictionary types. Basically we just allow merging any arrow data type
that is the same delta type

# Related Issue(s)
Fixes #2298

# Documentation

<!---
Share links to useful documentation
--->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: schema evolution not coercing with large arrow types aersam/delta-rs
2 participants
@aersam @ion-elgreco