fix: make struct fields nullable in stats schema

[qinix]

Description

Currently only top level fields are mapped to nullable=true in stats schema. However, delta-spark generated stats may have null stats fields even if origin data fields are declared as NOT NULL. So stats for columns nested within a struct should also be mapped to nullable=true as well.

[rtyler]

hm interesting, would you be able to add a test case or example stats to the pull request that we can incorporate as a test to ensure that nullable nested fields continue to be ssupported

[qinix]

Sure, I'll try to add a test case.

BTW, here is a minimal pyspark snippet to reproduce this case.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.1
      /_/

Using Python version 3.10.13 (main, Aug 24 2023 12:59:26)
Spark context Web UI available at http://192.168.31.133:4040
Spark context available as 'sc' (master = local[*], app id = local-1711560099703).
SparkSession available as 'spark'.

>>> from pyspark.sql.types import *; from delta.tables import *; import deltalake;

>>> schema = StructType([StructField("s", dataType = StructType([StructField("l", LongType(), nullable = False), StructField("b", BooleanType(), nullable = False)]), nullable = False)])

>>> DeltaTable.createOrReplace(spark).location('/tmp/test_delta_notnull_struct').addColumns(schema).execute()
<delta.tables.DeltaTable object at 0x1596c7ee0>

>>> spark.createDataFrame([{'s': {'l': 10, 'b': True}}, {'s': {'l': 20, 'b': False}}], schema).write.format('delta').mode('append').save('/tmp/test_delta_notnull_struct')

>>> deltalake.DeltaTable('/tmp/test_delta_notnull_struct')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/qinix/.virtualenvs/matrix/lib/python3.10/site-packages/deltalake/table.py", line 408, in __init__
    self._table = RawDeltaTable(
Exception: Json error: whilst decoding field 'minValues': whilst decoding field 's': Encountered unmasked nulls in non-nullable StructArray child: Field { name: "b", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }