[Python] csv does not work with a cedilla delimiter

[rkarthik29]

Describe the bug, including details regarding any error messages, version, and platform.

we are trying to parse a CSV file that has a cedilla \x03 delimiter. When passing this to pyarrow.csv.read_csv, the parse complains that the delimiter \x03 is not a valid ascii. Is there a way around this issue ?

ile ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:422, in pyarrow._csv.ParseOptions.init()

File ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:445, in pyarrow._csv.ParseOptions.delimiter.set()

File ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:46, in pyarrow._csv._single_char()

ValueError: Expecting an ASCII character

i get the above error if i use the character as delim "Ç"

but if i use \x03 , the data is read, but it is not parsed into columns.

i also tried "Ç".encode(ascii) and i get an error that the ascii code \xc7 is not valid. I tried chr(199). Nothing worked. Could anybody guide me in the right direction here. Dont want to use pandas

Component(s)

Python

[AlenkaF]

I think the issue here is that letter C with cedilla is not included in the main ASCII Table and I am not sure how to work around it.

I am not sure pandas will work here either.

[rkarthik29]

@AlenkaF It works with pandas if we use the unicode value \x03 for the cedilla as our seperator.

[rkarthik29]

reader=pd.read_csv(pfs.open("xyz.dat","rb")
,sep="\Ç",header=None,names=names,chunksize=100,encoding="utf-8",quotechar='"')

this works for pandas

[jorisvandenbossche]

There is currently no way around this: arrow only supports single ASCII characters as separator at the moment.

Related issues: #32432, #26411

Going to close this as duplicate of those other issues.