Can't optimize a table created by Spark.

[wjones127]

Environment

Delta-rs version:

Binding:

Environment:

• Cloud provider:
• OS:
• Other:

---

Bug

What happened:

https://github.com/delta-io/delta-rs/actions/runs/6243973257/job/16950202833?pr=1602

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: NoMetadata', rust\src\operations\optimize.rs:1381:18

What you expected to happen:

How to reproduce it:

        #[tokio::test]
        async fn works_on_spark_table() {
            // Create a temporary directory
            let tmp_dir = tempdir::TempDir::new("optimize-spark").unwrap();
            let table_uri = tmp_dir.path().to_str().to_owned().unwrap();

            // Copy recursively from the test data directory to the temporary directory
            let source_path = "tests/data/delta-1.2.1-only-struct-stats";
            fs_extra::dir::copy(source_path, tmp_dir.path(), &Default::default()).unwrap();

            // Run optimize
            let (_, metrics) = DeltaOps::try_from_uri(table_uri)
                .await
                .unwrap()
                .optimize()
                .await
                .unwrap();

            // Verify it worked
            assert_eq!(metrics.num_files_added, 1);
        }

More details:

[rtyler]

I saw this, I didn't realize it was a regression and thought it was related to the code in that pull request 🤔

@wjones127 are you going to be fixing this or is it "up for grabs"? 😄

[wjones127]

TBH I have no idea why this was in that PR. It doesn't have to do with upgrading PyArrow.

Up for grab for now, as I'll try to fix #1602 first.

[rtyler]

So the optimize actually fails but not for that reason. The reproduction case has to be modified slightly (which I have in my branch), after which the error is:

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: SchemaMismatch { msg: "Unexpected Arrow schema: g
ot: Field { name: \"integer\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"null\", data_type: Boolean, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Fi
eld { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"decimal\", data_type: Decimal128(8, 5), nulla
ble: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
 Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"date\", data_type: Date32, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata:
 {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { na
me: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metada
ta: {} }, Field { name: \"array\", data_type: List(Field { name: \"element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Struct([Field { n
ame: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, me
tadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", data_type: Struct([Field { name: \"struct_eleme
nt\", data_type: List(Field { name: \"element\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict
_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: fal
se, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", data_type: Int32, nullable: true, dict_id: 0,
 dict_is_ordered: false, metadata: {} }, expected: Field { name: \"integer\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field {
name: \"null\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_i
d: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name:
 \"decimal\", data_type: Decimal128(8, 5), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dic
t_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { nam
e: \"date\", data_type: Date32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Microsecond, None), nul
lable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { n
ame: \"entries\", data_type: Struct([Field { name: \"keys\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\",
data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"array\", data_type: List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"
struct_element\", data_type: Struct([Field { name: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable
: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", d
ata_type: Struct([Field { name: \"struct_element\", data_type: List(Field { name: \"item\", data_type: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"keys
\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", da
ta_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }" }', rust/src/operations/optimize.rs:1386:18

which looks much more like a real problem

[ion-elgreco]

So the optimize actually fails but not for that reason. The reproduction case has to be modified slightly (which I have in my branch), after which the error is:

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: SchemaMismatch { msg: "Unexpected Arrow schema: g
ot: Field { name: \"integer\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"null\", data_type: Boolean, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Fi
eld { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"decimal\", data_type: Decimal128(8, 5), nulla
ble: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
 Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"date\", data_type: Date32, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata:
 {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { na
me: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metada
ta: {} }, Field { name: \"array\", data_type: List(Field { name: \"element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Struct([Field { n
ame: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, me
tadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", data_type: Struct([Field { name: \"struct_eleme
nt\", data_type: List(Field { name: \"element\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict
_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: fal
se, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", data_type: Int32, nullable: true, dict_id: 0,
 dict_is_ordered: false, metadata: {} }, expected: Field { name: \"integer\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field {
name: \"null\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_i
d: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name:
 \"decimal\", data_type: Decimal128(8, 5), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dic
t_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { nam
e: \"date\", data_type: Date32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Microsecond, None), nul
lable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { n
ame: \"entries\", data_type: Struct([Field { name: \"keys\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\",
data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"array\", data_type: List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"
struct_element\", data_type: Struct([Field { name: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable
: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", d
ata_type: Struct([Field { name: \"struct_element\", data_type: List(Field { name: \"item\", data_type: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"keys
\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", da
ta_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }" }', rust/src/operations/optimize.rs:1386:18

which looks much more like a real problem

The arrow schema that is being written has nanoseconds timestamps while microseconds is expected in the delta schema. Did somewhere the default got changed in pyarrow to NS unit?

Also these errors could probably be easier to read if only the difference is shown instead of the full two schemas

[rtyler]

I hadn't had a chance to look deeper into this, good eye @ion-elgreco ! That means this is actually a test case for #1286! The reporter of that issue will be so happy to see this being worked on!

[ion-elgreco]

I hadn't had a chance to look deeper into this, good eye @ion-elgreco ! That means this is actually a test case for #1286! The reporter of that issue will be so happy to see this being worked on!

Is there something similar on the rust side for this: https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetReadOptions.html#pyarrow.dataset.ParquetReadOptions.coerce_int96_timestamp_unit

This is at least what I used to read spark-delta tables using Python bindings in delta-rs

[ion-elgreco]

@rtyler people can avoid this error by writing tables with this spark setting. The annoying default is INT96.. spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")