.get_add_actions() returns wrong column statistics when dataSkippingNumIndexedCols property of the table was changed

[robertkossendey]

Environment

Delta-rs version:
0.7.0

Binding:
Python

---

Bug

What happened:
.get_add_actions() returns wrong column statistics when dataSkippingNumIndexedCols property of the table was changed.
It returns only the stats for columns from the latest file.

What you expected to happen:
It should return all the stats for all columns of all the files.

How to reproduce it:

Had to do it in delta-spark, since I don't think there is a way of setting table properties in delta-rs.

path = f"./latestversion"

data = [
    (1, "a", None),
    (2, "b", "b"),
    (3, "c", "c"),
]
df = spark.createDataFrame(
    data,
    [
        "col1",
        "col2",
        "col3",
    ],
)
    
df.coalesce(1).write.mode("append").format("delta").save(path)

# .get_add_actions() will return min, max and null_count for col1, col2 and col3

spark.sql("ALTER TABLE delta.`FULL_PATH` SET TBLPROPERTIES(delta.dataSkippingNumIndexedCols = 1);")

df.coalesce(1).write.mode("append").format("delta").save(path)

# .get_add_actions() will only return min, max and null_count for col1

[mrjoe7]

This is caused by how DeltaTableState::stats_as_batch is producing aggregates here.
In your example above, the values of column max.col3 would look like ["c", None] which

.collect::<Option<Vec<&serde_json::Value>>>()

will convert into None because according to Option::from_iter api docs:

Takes each element in the Iterator: if it is None, no further elements are taken, and the None is returned.

[wjones127]

It should return all the stats for all columns of all the files.

@robertkossendey do you want it to return statistics for all columns? Or just all columns that at some point collected statistics? I agree the current behavior is wrong, but I'm just not sure we want to return statistics for columns that are always null.

I could see an argument that it's nice to have all the columns represented there. But there may be some users who have tables with very large schemas, where they intentionally don't collect statistics for most columns.