[Spark] Support write partition columns to data files for Delta

This support write partition columns to data files for Delta. The feature is required by Uniform Iceberg as Iceberg spec defines so. The approach is to copy FileFormatWriter from Spark as a DeltaFileFormatWriter, and add the option and logic there to support writing partition columns. Closes #2367 GitOrigin-RevId: 77657bb422ce93b924f3cb25548e845477f8632f
delta-io · Dec 20, 2023 · d4fd5e2 · d4fd5e2
1 parent 75dba07
commit d4fd5e2
Show file tree

Hide file tree

Showing 3 changed files with 490 additions and 4 deletions.
diff --git a/spark/src/main/scala/org/apache/spark/sql/delta/DeltaOptions.scala b/spark/src/main/scala/org/apache/spark/sql/delta/DeltaOptions.scala
@@ -289,6 +289,10 @@ object DeltaOptions extends DeltaLogging {
    */
   val STREAMING_SOURCE_TRACKING_ID = "streamingSourceTrackingId"
 
+  /**
+   * An option to control if delta will write partition columns to data files
+   */
+  val WRITE_PARTITION_COLUMNS = "writePartitionColumns"
 
   val validOptionKeys : Set[String] = Set(
     REPLACE_WHERE_OPTION,
@@ -323,7 +327,8 @@ object DeltaOptions extends DeltaLogging {
     "checkpointLocation",
     "path",
     VERSION_AS_OF,
-    TIMESTAMP_AS_OF
+    TIMESTAMP_AS_OF,
+    WRITE_PARTITION_COLUMNS
   )