append is deleting records

[machov]

Environment

append() is deleting records

Any idea why?

Delta-rs version: 0.18.2

Binding:

Environment:

• Cloud provider: Azure
• OS: Linux
• Other: Python version 3.10.12

---

Bug

What happened:

The following write_deltalake function is deleting records and not only appending

import pyarrow as pa

        pa_table = pa.Table.from_pandas(validated_df, schema=merged_schema, preserve_index=False)
        write_deltalake(
            abfss_table_path,
            pa_table,
            mode="append",
            schema_mode="merge",
            engine="rust",
            storage_options=storage_options,
        )

What you expected to happen:

I'd expect row count to increase, however row count decreased significantly!

How to reproduce it:

This is a very large table with 300k+ records

Resulting delta table became 30k+ records, deleting hundreds of thousands

More details:

[ion-elgreco]

@machov I can look into it if you give a reproducible example

[machov]

good point, let me look into that, closing for now,