fix: schema evolution not coercing with large arrow types

[aersam]

Description

This fixes schema merge for large arrow types in combination with dictionary types. Basically we just allow merging any arrow data type that is the same delta type

Related Issue(s)

Fixes #2298

Documentation

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[ion-elgreco]

@aersam wouldn't it maybe be better to have the schema merge function take in delta fields instead of arrow

[aersam]

@aersam wouldn't it maybe be better to have the schema merge function take in delta fields instead of arrow

Yes! But thats a topic throughout the whole crate, everything is based on arrow schemas instead of delta schemas.

[ion-elgreco]

@aersam can be addressed in a follow up if it's to big of a change now.

Btw, forgot to add that in the issue but it is also not working with large lists, and nested versions

[aersam]

@aersam can be addressed in a follow up if it's to big of a change now.

Btw, forgot to add that in the issue but it is also not working with large lists, and nested versions

To merge nested correctly, there's no other way than to really do the full thing on our own. So that's what I did now. Since we're still merging arrow schemas, I now do first a conversion to delta and then back to arrow again, which is not optimal, but still does work.

[aersam]

I'll adjust the tests 🙂

[aersam]

Should I move the delta schema merge code to kernel?

[ion-elgreco]

@aersam I believe kernel should only for stuff related to delta protocol implementation, and schema evolution is more on the operation side

[aersam]

Oh I guess I should merge metadata, too. Will add that

[ion-elgreco]

@aersam Great fix!! 😃