[SPARK-45538][PYTHON][CONNECT] pyspark connect overwrite_partitions bug

### What changes were proposed in this pull request? Fix a bug in pyspark connect. DataFrameWriterV2.overwritePartitions set mode as overwrite_partitions [pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as overwrite_partition [pyspark/sql/connect/plan.py, line 1660] ### Why are the changes needed? make dataframe.writeTo(table).overwritePartitions() work ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No test. This bug is very obvious. ### Was this patch authored or co-authored using generative AI tooling? No Closes #43367 from xieshuaihu/python_connect_overwrite. Authored-by: xieshuaihu <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
apache · Oct 16, 2023 · 9bdad31 · 9bdad31
1 parent 7796d8a
commit 9bdad31
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/python/pyspark/sql/connect/plan.py b/python/pyspark/sql/connect/plan.py
@@ -1743,7 +1743,7 @@ def command(self, session: "SparkConnectClient") -> proto.Command:
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_CREATE
             elif wm == "overwrite":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_OVERWRITE
-            elif wm == "overwrite_partition":
+            elif wm == "overwrite_partitions":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_OVERWRITE_PARTITIONS
             elif wm == "append":
                 plan.write_operation_v2.mode = proto.WriteOperationV2.Mode.MODE_APPEND