fix(python): fixed large_dtype to schema convert

[sherlockbeard]

Description

added large_dtype to schema convert

closes #2374

Related Issue(s)

Documentation

[chitralverma]

@rtyler facing the same issue with merge instead of write.
can this option be added to merge as well ?

[ion-elgreco]

@chitralverma what exactly are you facing?

[hitesh1997]

@ion-elgreco
The issue arises when we try to perform the merge operation like below -

(
    df.write_delta(
        delta_table_path,
        mode="merge",
        delta_merge_options={
            "predicate": "s.col1 = t.col1",
            "source_alias": "s",
            "target_alias": "t",
        },
        storage_options=delta_storage_options,
    )
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
)

Here delta_storage_options is this -

delta_storage_options = {
    "aws_access_key_id": aws_key_id,
    "aws_secret_access_key": aws_secret_key,
    "aws_endpoint": aws_endpoint,
    "AWS_S3_ALLOW_UNSAFE_RENAME": "true",
}

The above code works when trying to run this on local without mentioning storage_options.

When we try to run this on aws, we get this error message -

Error writing to Delta Table: Generic DeltaTable error: **type_coercion**
caused by
Error during planning: Failed to coerce then ([LargeList(Field { name: "item", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), LargeList(Field { name: "item", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (None) to common types in CASE WHEN expression

Our delta table schema is like this below. Which contain columns of type "List" -

schema = pa.schema([
    ('col1', pa.string()),
    ('col2', pa.string()),
    ('col3', pa.string()),
    ('col4', pa.string()),
    ('col5', pa.string()),
    ('col6', pa.string()),
    ('col7', pa.string()),
    ('col8', pa.string()),
    ('col9', pa.string()),
    ('col10', pa.list_(pa.string())),
    ('col11', pa.list_(pa.string())),
    ('col12', pa.string()),
    ('col13', pa.float64()),
])

[ion-elgreco]

@hitesh1997 please have a look at this issue: #2851

[hitesh1997]

Hi @ion-elgreco,

The issue was fixed by downgrading to deltalake==0.18.2 from 0.19.1

[ion-elgreco]

@hitesh1997 0.18.2 just buries the issue by always downcasting to normal types, you could do the same thing in 0.20 already by manually providing large_dtypes = False