[python] Different stringification of partition values in reader and writer

[giacomorebecchi]

Environment

Delta-rs version: python v0.10.2

---

Bug

The delta reader and writer treat differently partition values which are not strings. You can clearly see it from:

delta-rs/python/deltalake/table.py

Lines 614 to 627 in a74589b

	def __stringify_partition_values(
	self, partition_filters: Optional[List[Tuple[str, str, Any]]]
	) -> Optional[List[Tuple[str, str, Union[str, List[str]]]]]:
	if partition_filters is None:
	return partition_filters
	out = []
	for field, op, value in partition_filters:
	str_value: Union[str, List[str]]
	if isinstance(value, (list, tuple)):
	str_value = [str(val) for val in value]
	else:
	str_value = str(value)
	out.append((field, op, str_value))
	return out

delta-rs/python/deltalake/writer.py

Lines 494 to 509 in a74589b

	def __encode_partition_value(val: Any) -> str:
	# Rules based on: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#partition-value-serialization
	if isinstance(val, bool):
	return str(val).lower()
	if isinstance(val, str):
	return val
	elif isinstance(val, (int, float)):
	return str(val)
	elif isinstance(val, date):
	return val.isoformat()
	elif isinstance(val, datetime):
	return val.isoformat(sep=" ")
	elif isinstance(val, bytes):
	return val.decode("unicode_escape", "backslashreplace")
	else:
	raise ValueError(f"Could not encode partition value for type: {val}")

An easy fix would be to make the second function accessible, removing the double underscore, and using it inside the reader.

[wjones127]

Could you describe an actual error or failure you have encountered as a result of this? Or even a reproducible example?

[giacomorebecchi]

Could you describe an actual error or failure you have encountered as a result of this? Or even a reproducible example?

The following is a MRE that displays that, due to the different transformation of the partition value in the writer and the reader, the file_uris method fails to find the correct file uris. I used the boolean type to display the error, because in the writer you call str(value).lower(), but the same MRE would work for any other type which has a different implementation. Note in the MRE that, since the integers are casted with str(value) in both the reader and the writer. Also, note the fact that the .to_pyarrow_table() does not suffer from this bug.

import deltalake
import pyarrow as pa

fp = "./resources/mytable"

ta = pa.Table.from_pydict(
    {
        "bool_col": [True, False, True, False],
        "int_col": [0, 1, 2, 3],
        "str_col": ["a", "b", "c", "d"],
    }
)

deltalake.write_deltalake(fp, ta, partition_by=["bool_col", "int_col"], mode="overwrite")

dt = deltalake.DeltaTable(fp)

assert dt.to_pyarrow_table(
    filters=[
        ("int_col", "=", 0),
        ("bool_col", "=", True),
    ]
).num_rows == 1

assert len(dt.file_uris(
    partition_filters=[
        ("int_col", "=", 0),
        ("bool_col", "=", "true"),
    ]
)) == 1 # finds the actual partition because it looks for "bool_col=true"

assert len(dt.file_uris(
    partition_filters=[
        ("int_col", "=", 0),
        ("bool_col", "=", True),
    ]
)) == 1  # does not find any partition because it looks for "bool_col=True"