Transaction log JSON formatting issue when writing data via Python bindings

[tobyych]

hi there, i am using deltalake 0.6.4 python bindings to try writing delta tables to a S3-compatible storage.

i noticed an issue that the transaction log generated is not entirely the same as the example in https://github.com/delta-io/delta/blob/master/PROTOCOL.md (refer to the image attached, left one is the log generated by deltalake package, right one is the example on GitHub)

this caused the data query engine Trino (version: 397)
not able to read the delta log properly.

the relevant part of the error message is:
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type java.lang.String from Array value (token JsonToken.START_ARRAY)
at [Source: (String)"{"commitInfo":{"clientVersion":"delta-rs.0.5.0","operation":"delta-rs.Write","operationParameters":{"mode":"Append","partitionBy":[],"predicate":null},"timestamp":1670980236192}}"; line: 1, column: 131] (through reference chain: io.trino.plugin.deltalake.transactionlog.DeltaLakeTransactionLogEntry["commitInfo"]->io.trino.plugin.deltalake.transactionlog.CommitInfoEntry["operationParameters"]->java.util.LinkedHashMap["partitionBy"])

may i know if there is any method to work around the issue?

Slack Message

[rtyler]

As I mentioned in Slack, I think it's not that big of a deal for us to add some quotes around the partitionBy. I think the Trinio plugin also has a bug here because commitInfo is optional in the protocol and therefore I don't believe it should be considered to have a required and agreed upon schema

[sfilimonov-exos]

delta-spark also fails to read such a table with unquoted partitionBy.

Code to reproduce:

import pandas as pd
from delta import DeltaTable
from delta import configure_spark_with_delta_pip
from deltalake.writer import write_deltalake
from pyspark.sql import SparkSession

spark = configure_spark_with_delta_pip(SparkSession.builder.appName("lo").master("local[*]")) \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

table = "table_12345"

df = pd.DataFrame(data=[("one", 2), ("two", 2)], columns=["name", "val"])

write_deltalake(table, df)
# First write doesn't add partitionBy to commit info - thus need to write one more time 
write_deltalake(table, df, mode="append")

dt = DeltaTable.forPath(spark, table)
dt.toDF().show()  # The table itself can be read without an issue.
dt.history().show()  # But not the history

delta-spark 2.2.0
deltalake 0.6.4

22/12/16 11:43:03 ERROR Executor: Exception in task 9.0 in stage 6.0 (TID 62)
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Array value (token `JsonToken.START_ARRAY`)
 at [Source: (String)"{"commitInfo":{"clientVersion":"delta-rs.0.5.0","operation":"delta-rs.Write","operationParameters":{"mode":"Append","partitionBy":[],"predicate":null},"timestamp":1671183764077}}"; line: 1, column: 131] (through reference chain: org.apache.spark.sql.delta.actions.SingleAction["commitInfo"]->org.apache.spark.sql.delta.actions.CommitInfo["operationParameters"]->com.fasterxml.jackson.module.scala.deser.GenericMapFactoryDeserializerResolver$BuilderWrapper["partitionBy"])