[SPARK-43363][SQL][PYTHON] Make to call astype to the category type…

… only when the arrow type is not provided ### What changes were proposed in this pull request? Makes to call `astype` to the category type only when the arrow type is not provided. ### Why are the changes needed? Now that the minimum version of pyarrow is `1.0.0`, a workaround for pandas' categorical type for pyarrow can be removed if the arrow type is provided. > Note: This can be removed once minimum pyarrow version is >= 0.16.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closes #41041 from ueshin/issues/SPARK-43363/categorical_type. Authored-by: Takuya UESHIN <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
apache · May 5, 2023 · 7b5d7af · 7b5d7af
1 parent cb26ad8
commit 7b5d7af
Showing 1 changed file with 1 addition and 2 deletions.
diff --git a/python/pyspark/sql/pandas/serializers.py b/python/pyspark/sql/pandas/serializers.py
@@ -226,8 +226,7 @@ def create_array(s, t):
                 s = _check_series_convert_timestamps_internal(s, self._timezone)
             elif t is not None and pa.types.is_map(t):
                 s = _convert_dict_to_map_items(s)
-            elif is_categorical_dtype(s.dtype):
-                # Note: This can be removed once minimum pyarrow version is >= 0.16.1
+            elif t is None and is_categorical_dtype(s.dtype):
                 s = s.astype(s.dtypes.categories.dtype)
             try:
                 array = pa.Array.from_pandas(s, mask=mask, type=t, safe=self._safecheck)