UnicodeDecodeError on Windows: charmap codec

[ryanmeekins]

Describe the bug
I'm trying to run sqlfmt . on my dbt repo but am getting a UnicodeDecodeError. I can't figure out what file sqlfmt is getting stuck on.

To Reproduce
I'm not sure how others can reproduce until I know what file is an issue. I've tried sqlfmt -v . but don't see any additional logs.

Expected behavior
I expect sqlfmt to finish processing all files.

Actual behavior

$ sqlfmt --verbose .
 59%|█████████████████████████▏                 | 745/1269 [00:08<00:27, 19.28file/s]concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\concurrent\futures\process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\api.py", line 221, in _format_one
    source = _read_path_or_stdin(path)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\api.py", line 258, in _read_path_or_stdin
    source = f.read()
  File "C:\Program Files\Python310\lib\encodings\cp1252.py", line 23, in decode       
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2581: character maps to <undefined>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\Scripts\sqlfmt.exe\__main__.py", line 7, in <module>
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\cli.py", line 181, in sqlfmt
    report = api.run(files=matched_files, mode=mode, callback=progress_callback)      
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\api.py", line 76, in run
    results = _format_many(files, cache, mode, callback=callback)
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\api.py", line 184, in _format_many
    asyncio.run(_multiprocess_map(format_func, paths, callback=callback))
  File "C:\Program Files\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 649, in run_until_complete
    return future.result()
  File "C:\Users\ryan.meekins\Desktop\transformer\.venv\lib\site-packages\sqlfmt\api.py", line 212, in _multiprocess_map
    results: List[R] = await asyncio.gather(*tasks)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2581: character maps to <undefined>

Additional context
sqlfmt, version 0.16.0
Python 3.10.10

[ryanmeekins]

Just noticed that sqlfmt . does work on our project when running in a linux environment (ubuntu-20.04). Realized this after setting up a GitHub Action. Locally, I'm running in a 64-bit Windows 10 Python environment.

[tconbeer]

I think this is the same issue as #350. Can you try the fix in that issue and let me know if that solves it?

[ryanmeekins]

that solved it, thanks!