[SPARK-45538][PYTHON][CONNECT] pyspark connect overwrite_partitions bug

Fix a bug in pyspark connect. DataFrameWriterV2.overwritePartitions set mode as overwrite_partitions [pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as overwrite_partition [pyspark/sql/connect/plan.py, line 1660] make dataframe.writeTo(table).overwritePartitions() work No No test. This bug is very obvious. No Closes #43367 from xieshuaihu/python_connect_overwrite. Authored-by: xieshuaihu <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 9bdad31) Signed-off-by: Hyukjin Kwon <[email protected]>
apache · Oct 16, 2023 · daa3281 · daa3281
1 parent 146fba1
commit daa3281
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/python/pyspark/sql/connect/plan.py b/python/pyspark/sql/connect/plan.py
@@ -1655,7 +1655,7 @@ def command(self, session: "SparkConnectClient") -> proto.Command:
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_CREATE
             elif wm == "overwrite":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_OVERWRITE
-            elif wm == "overwrite_partition":
+            elif wm == "overwrite_partitions":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_OVERWRITE_PARTITIONS
             elif wm == "append":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_APPEND