Generic DeltaTable error: type_coercion in Struct column in merge operation

[gustavodecarlo]

Environment

Delta-rs version: rust-v0.16.5

Binding: python-v0.14.0

Environment: local machine

• Cloud provider:
• OS: macOS Sonoma 14.2.1
• Other: python version 3.9.13

---

Bug

What happened:
_internal.DeltaError: Generic DeltaTable error: type_coercion
caused by
Error during planning: Failed to coerce then ([Struct([Field { name: "created_datetime", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "version", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), Struct([Field { name: "created_datetime", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "version", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), Struct([Field { name: "created_datetime", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "version", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), Struct([Field { name: "created_datetime", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "version", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), Struct([Field { name: "created_datetime", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "version", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])]) and else (None) to common types in CASE WHEN expression

What you expected to happen:

merge executed successfully

How to reproduce it:
Need to pip install both polars and deltalake
Same code in this repo: https://github.com/gustavodecarlo/poc-python-deltars-upsert/blob/main/poc_python_deltars_upsert/dataflow.py#L12

from deltalake import DeltaTable, Schema
import polars as pl

df = pl.DataFrame(
        {
            "sales_order_id": ["1000", "1001", "1002", "1003"],
            "product": ["bike", "scooter", "car", "motorcycle"],
            "order_date": [
                datetime(2023, 1, 1),
                datetime(2023, 1, 5),
                datetime(2023, 1, 10),
                datetime(2023, 2, 1),
            ],
            "sales_price": [120.25, 2400, 32000, 9000],
            "paid_by_customer": [True, False, False, True],
            "metadata": [
                {
                    "created_datetime": "2023-01-10",
                    "version": "1"
                },
                {
                    "created_datetime": "2023-01-12",
                    "version": "1"
                },
                {
                    "created_datetime": "2023-01-10",
                    "version": "1"
                },
                {
                    "created_datetime": "2023-01-13",
                    "version": "1"
                }
            
            ]
        }
    )
    print(df)

    df.write_delta("data/sales_orders", mode="append")

    new_data = pl.DataFrame(
        {
            "sales_order_id": ["1002", "1004"],
            "product": ["car", "car"],
            "order_date": [datetime(2023, 1, 10), datetime(2023, 2, 5)],
            "sales_price": [30000.0, 40000.0],
            "paid_by_customer": [True, True],
            "metadata": [
                {
                    "created_datetime": "2023-01-10",
                    "version": "1"
                },
                {
                    "created_datetime": "2023-01-13",
                    "version": "1"
                }
            ]
        }
    )

    dt = DeltaTable("data/sales_orders")
    source = new_data.to_arrow()
    delta_schema = Schema.from_pyarrow(source.schema).to_pyarrow()
    source = source.cast(delta_schema)

    (
        dt.merge(
            source=source,
            predicate="s.sales_order_id = t.sales_order_id",
            source_alias="s",
            target_alias="t",
        )
        .when_matched_update_all()
        .when_not_matched_insert_all()
        .execute()
    )

    print(pl.read_delta("data/sales_orders"))

[ion-elgreco]

Struct type coercion is not yet built in data fusion sadly.

I will expose the large_dtypes parameter so you can control that and set it to False, then it should work

[gustavodecarlo]

Struct type coercion is not yet built in data fusion sadly.

I will expose the large_dtypes parameter so you can control that and set it to False, then it should work

Where i'm set this config? @ion-elgreco

[ion-elgreco]

@gustavodecarlo it will be on .merge()

[gustavodecarlo]

@gustavodecarlo it will be on .merge()

Let me try. thanks for the insight

[ion-elgreco]

@gustavodecarlo it's not yet available. I will make it available in the next release though

[gustavodecarlo]

@gustavodecarlo it's not yet available. I will make it available in the next release though

Thanks for the information let's wait. And this issue can we close?

[gustavodecarlo]

@ion-elgreco, worked with large_dtypes=False. Thanks for the help.