UnicodeError on path walking

[agjohnson]

Sep 13 15:59:07 build03 readthedocs/readthedocs.doc_builder.environments[19039]: ERROR (Build) [pipenv:latest] 'ascii' codec can't decode byte 0xf0 in position 15: ordinal not in range(128) [readthedocs.doc_builder.environments:317]
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/readthedocs/projects/tasks.py", line 154, in run_setup
    self.config = load_yaml_config(version=self.version)
  File "/home/docs/checkouts/readthedocs.org/readthedocs/doc_builder/config.py", line 148, in load_yaml_config
    env_config=sphinx_env_config,
  File "/home/docs/local/lib/python2.7/site-packages/readthedocs_build/config/config.py", line 362, in load
    config_files = list(find_all(path, CONFIG_FILENAMES))
  File "/home/docs/local/lib/python2.7/site-packages/readthedocs_build/config/find.py", line 6, in find_all
    for root, dirs, files in os.walk(path, topdown=True):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/docs/lib/python2.7/os.py", line 284, in walk
    if isdir(join(top, name)):
  File "/home/docs/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 15: ordinal not in range(128)

[kennethreitz]

hmmmmm

[stsewd]

@agjohnson I think I found the problem. I did some reading here https://stackoverflow.com/questions/5974585/python-not-able-to-open-file-with-non-english-characters-in-path

On python2 the path should be passed as a byte-string, no unicode. On rtd this is passed as unicode. I did some local testing with this project https://bitbucket.org/randrian/opengpias from readthedocs/readthedocs.org#3732.

# this fails
find_one(
    u'/home/stsewd/rtd/readthedocs.org/user_builds/hg/checkouts/latest/',
    ('readthedocs.yml', '.readthedocs.yml')
)

# this pass
find_one(
    str(u'/home/stsewd/rtd/readthedocs.org/user_builds/hg/checkouts/latest/'),
    ('readthedocs.yml', '.readthedocs.yml')
)

Should I submit this fix on this repo (forcing to str the path variable before it's passed to find_one) or on the rtd repo (passing the path variable as a bytestring)?

[ericholscher]

Writing a test to reproduce the error first would be a good start. I think this might be tricky to do for py2/3 at the same time, but +1 on fixing it!

[stsewd]

@ericholscher It was more difficult than I thought, but I think I did it 😄 #45