Python write_deltalake with schema_mode="merge" casts types

[robjhornby-ts]

Environment

Delta-rs version: 0.18.1 (Python deltalake 0.18.1)

Binding:

Environment:

• Cloud provider:
• OS: MacOS Sonoma 14.5 (Intel)
• Other:

---

Bug

What happened:

Using write_deltalake to append data to a table with schema_mode="merge", if I change the type of a field in a struct, the data is cast to the existing table schema (e.g. a string "1" becomes an int 1) instead of raising an error because I'm trying to merge incompatible schemas.

What you expected to happen:

When a field's type changes, I'd expect the merge to fail. (Unless I deliberately configure something like type widening int32 to int64 etc.)

How to reproduce it:

Here, a struct contains a field "x" which changes from int64 to string, the table schema is printed before and after, and the final data shows the string has been cast to an int

import pyarrow as pa
from pathlib import Path
import tempfile
from deltalake import DeltaTable, write_deltalake


def invalid_schema_merge():
    with tempfile.TemporaryDirectory() as tmpdir:
        data_path = Path(tmpdir)
        schema_before = pa.schema(
            pa.struct(
                [
                    pa.field("x", pa.int64()),
                ]
            )
        )
        schema_after = pa.schema(
            pa.struct(
                [
                    pa.field("x", pa.string()),
                ]
            )
        )
        data_before = pa.Table.from_pylist([{"x": 100}], schema=schema_before)
        data_after = pa.Table.from_pylist([{"x": "1"}], schema=schema_after)

        table = DeltaTable.create(
            data_path,
            schema=schema_before,
            mode="overwrite",
        )
        print(table.schema())
        write_deltalake(
            table,
            data_before,
            schema=schema_before,
            mode="append",
            schema_mode="merge",
            engine="rust",
        )
        print("-" * 80)
        print(table.schema())

        write_deltalake(
            table,
            data_after,
            schema=schema_after,
            mode="append",
            schema_mode="merge",
            engine="rust",
        )
        print("-" * 80)
        print(table.schema())
        print("-" * 80)
        print("Final data: ", table.to_pyarrow_table())


invalid_schema_merge()

Prints:

Schema([Field(x, PrimitiveType("long"), nullable=True)])
--------------------------------------------------------------------------------
Schema([Field(x, PrimitiveType("long"), nullable=True)])
--------------------------------------------------------------------------------
Schema([Field(x, PrimitiveType("long"), nullable=True)])
--------------------------------------------------------------------------------
Final data:  pyarrow.Table
x: int64
----
x: [[1],[100]]

The schema is the same after writing the new data containing a string, and the string has been cast to an int

I don't know whether this is a bug or a feature request. I see some mention of type casting in the code but couldn't find it in docs.

I'd like to be able to merge schemas in a way which will raise an exception if types have changed, unless I enable a setting for type casting (or similar). Let me know if more info would be useful, or if you want me to check more cases

[ion-elgreco]

This should indeed raise, since the types are not the same

[ion-elgreco]

@robjhornby-ts as a side effect this incorrect behaviour is fixed with the changes I made in casting logic. It will be available in v0.19