[BUG] Incompatibility with deltalake v0.19.0 and above - failure in write_deltalake() #2753

[igor-pechersky]

#2753

[igor-pechersky]

this PR conflicts with #2704 and overrides it

[codspeed-hq]

CodSpeed Performance Report

Merging #2754 will degrade performances by 57.59%

_{Comparing igor-pechersky:bugfix/deltalake-2753 (2e96ee5) with main (a97d871)}

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 14 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	igor-pechersky:bugfix/deltalake-2753	Change
❌	test_count[1 Small File]	10.1 ms	23.7 ms	-57.59%
⚡	test_show[100 Small Files]	55.8 ms	49.7 ms	+12.22%

[samster25]

hi @igor-pechersky! Thanks for contribution. Do you mind sharing the error you receiving prior to this PR?

Looks like we have a test cast failing because the data schema and table schema differ

Data schema:
a: int64
b: double
c: string
d: binary

vs

Table Schema:
a: int64
b: double
c: large_string
d: large_binary

We may need to still perform a manual cast

[igor-pechersky]

#2754 (comment)
hi @samster25

Do you mind sharing the error you receiving prior to this PR?

sure
#2753

We may need to still perform a manual cast

here
dc6c75b

[djouallah]

try large_dtypes=True when writing to delta

[igor-pechersky]

@djouallah
see delta-io/delta-rs@5e55cfb#diff-8cf13a20f8e3f8d6b75c9fa25af0a006bf9806623721d15b9bc5fe9f260ac3b2L20

[kevinzwang]

Hi @igor-pechersky I didn't realize that we already had a PR open for this issue before I made #2827 😄

Thanks for contributing this! However I think I would prefer to merge in my PR instead because it maintains compatibility with past deltalake versions.

[kevinzwang]

Closing this because we are merging in #2827!

[jaychia]

Thanks @igor-pechersky for taking a stab at this!