Internal Parquet panic when using a Map type.

[cmackenzie1]

Environment

Delta-rs version: 0.14

Binding: rust

Environment:

• Cloud provider:
• OS:
• Other:

---

Bug

What happened:

Writing JSON records with a Map(String, String) fails panics during parquet flush and close.

thread 'local::test_issue_parquet_bit_util' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`', /Users/cole/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-45.0.0/src/util/bit_util.rs:272:9

What you expected to happen:

Not to panic and error returned if applicable.

How to reproduce it:

let fields: Vec<SchemaField> = vec![SchemaField::new(
    "metadata".to_string(),
    SchemaDataType::map(SchemaTypeMap::new(
        Box::new(SchemaDataType::primitive("string".to_string())),
        Box::new(SchemaDataType::primitive("string".to_string())),
        true,
    )),
    true,
    HashMap::new(),
)];
let schema = deltalake::Schema::new(fields);
let table = deltalake::DeltaTableBuilder::from_uri("./tests/data/map-null").build()?;
let _ = DeltaOps::from(table)
    .create()
    .with_columns(schema.get_fields().to_owned())
    .await?;

let mut table = deltalake::open_table("./tests/data/map-null").await?;

let mut writer = JsonWriter::for_table(&table).unwrap();
let _ = writer
    .write(vec![
        serde_json::json!({"metadata": {"hello": "world", "something": null}}),
    ])
    .await
    .unwrap();
writer.flush_and_commit(&mut table).await.unwrap();

More details:

This value is true https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_arrow.rs#L170C21-L170C25 , where arrow defines it as false https://github.com/apache/arrow-rs/blob/master/arrow-schema/src/field.rs#L230. This is also described in apache/arrow-rs#1697.