SOLUTION IN COMMENTS: write_parquet() using pyarrow with a "partition_cols" of type "str" maps all partition values to "...". This is NOT an issue in pyarrow.parquet.write_to_dataset

[fpbeekhof1977]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
data = {'k':[10, 20], 'col1': [1,2], 'col2':[3,4], 'p':['a', 'b'] }
d = pl.DataFrame(data)
d.write_parquet('/tmp/d-polars', use_pyarrow=True, pyarrow_options={'partition_cols': ['p']})
import pyarrow as pa
pa.parquet.write_to_dataset(pa.Table.from_pydict(data), '/tmp/d-pyarrow', partition_cols=['p'])

Now try:

$ ls -lR /tmp/d-polars
'p=...'
$ ls -lR /tmp/d-pyarrow
'p=a'  'p=b'

Log output

No response

Issue description

All values of the partition column are mapped onto the string "..." rather than their value.

Expected behavior

Partition columns of string type are correctly written with their value intact.

Installed versions

--------Version info---------
Polars:               0.20.16
Index type:           UInt32
Platform:             Linux-5.15.0-1055-aws-x86_64-with-glibc2.35
Python:               3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2024.3.1
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.7.0
numpy:                1.23.5
openpyxl:             <not installed>
pandas:               1.5.3
pyarrow:              8.0.0
pydantic:             1.10.6
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[fpbeekhof1977]

The issue does not appear when creating the dataset directly in py-arrow.
However, it does happen when converting a dataframe to pyarrow and then using pyarrow's parquet.write_to_dataset method.

[fpbeekhof1977]

The reason is that polars' ".to_arrow()" converts "str" columns to pyarrow's "large_string" rather than "string".
The partitions are correctly written if the pyarrow type is "string" rather than "large_string".

SOLUTION: "to_arrow()" should map columns of type "str" onto pyarrow's "string", not "large_string".