Cannot merge into table partitioned by date type column on 0.16.3

[echai58]

Environment

Delta-rs version: 0.16.3

Binding: python

---

Bug

What happened:
After upgrading to 0.16.3 from 0.15.3, I'm seeing a DeltaError: Generic DeltaTable error: Unable to convert expression to string error when trying to merge into a table that is partitioned by a column of type pa.date32(). If I don't partition by date, or partition by a column of string/int type, the merge succeeds. So it is something to do with date type partitions.

What you expected to happen:
This used to work, so I would expect this behavior to not break in later versions. I'm not sure if this degradation came from 0.16.3 specifically, or some version in between 0.15.3 and now.

How to reproduce it:

import pyarrow as pa
import pandas as pd
from deltalake import DeltaTable

schema = pa.schema(
    [
        ("date", pa.date32()),
        ("k", pa.string()),
        ("v", pa.string()),
    ]
)

DeltaTable.create("test", schema=schema, partition_by=["date"])

t = pd.DataFrame.from_dict(
    {
        "date": [pd.to_datetime("2022-02-01")],
        "k": ["a"],
        "v": ["b"],
    }
)

dt.merge(
    t,
    predicate='s.date = t.date',
    source_alias="s",
    target_alias="t",
).when_matched_update_all().when_not_matched_insert_all().execute()

The above is a minimal reproduction, leading to the following error DeltaError: Generic DeltaTable error: Unable to convert expression to string.

More details:

[ion-elgreco]

@echai58 would you be able to check at what version this stopped working?

[echai58]

@echai58 would you be able to check at what version this stopped working?

@ion-elgreco Yes, just went through versions and it started breaking on 0.16.2, worked on versions before that.

[ion-elgreco]

@echai58 great, thanks for checking! I'll take a look in the next couple days, it may have to do with the early filter construction

[echai58]

@ion-elgreco thanks for working on a fix for this! Do you have plans to do a new version release with this fix soon? It's blocking me upgrading right now 🙂

[ion-elgreco]

@ion-elgreco thanks for working on a fix for this! Do you have plans to do a new version release with this fix soon? It's blocking me upgrading right now 🙂

Somewhere next week I guess

[echai58]

Hi @ion-elgreco, wanted to follow up and check if there are any updates on when 0.16.5 will be released?

[ion-elgreco]

Hi @echai58, I want to do a 0.17 release but with these PRs in as well:

• fix(rust): timestamp deserialization format, missing type #2383
• feat(rust): advance state in post commit #2396