UnicodeDecodeError is raised when requirements file contains multibyte characters

[Hiroshiba]

Environment Versions

1. OS Version: Windows 10
2. Python version: Python 3.8.10
3. pip version: pip 21.3.1 from c:\users\hihok\github\voicevox_engine\.venv\lib\site-packages\pip (python 3.8)
4. pip-tools version: pip-compile, version 6.4.0

Steps to replicate

1. Write a multibyte comment in the requirements.in file like # いつもありがとう、pip-tools！
2. pip-compile requirements.in

Expected result

Ignore multibyte comments and execute.

Actual result

$ pip-compile requirements.in
Traceback (most recent call last):
  File "C:\Users\hihok\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\hihok\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\Scripts\pip-compile.exe\__main__.py", line 7, in <module>
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\piptools\scripts\compile.py", line 417, in cli
    constraints.extend(
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\piptools\_compat\pip_compat.py", line 23, in parse_requirements
    for parsed_req in _parse_requirements(
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\req\req_file.py", line 145, in parse_requirements
    for parsed_line in parser.parse(filename, constraint):
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\req\req_file.py", line 326, in parse        
    yield from self._parse_and_recurse(filename, constraint)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\req\req_file.py", line 331, in _parse_and_recurse
    for line in self._parse_file(filename, constraint):
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\req\req_file.py", line 360, in _parse_file  
    _, content = get_file_content(filename, self._session)
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\req\req_file.py", line 533, in get_file_content
    content = auto_decode(f.read())
  File "C:\Users\hihok\Github\voicevox_engine\.venv\lib\site-packages\pip\_internal\utils\encoding.py", line 34, in auto_decode 
    return data.decode(
UnicodeDecodeError: 'cp932' codec can't decode byte 0x81 in position 84: illegal multibyte sequence

[Rogdham]

This is pip using your local encoding (cp932) to decode the file, whereas it that file is encoding differently (utf8).

Could you try to add a comment defining the encoding on top of the file?

For example, the following should work:

# coding=utf8

# いつもありがとう、pip-tools！
pip-tools

[Hiroshiba]

Thanks for the reply!
It worked fine with the method you showed me!