InstalledRepository load() breaks if there's an empty string for a env.sys_path variable

[ampersand-five]

• [x ] I am on the latest Poetry version.
• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• [x ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: <Ubuntu 18.04.3>
• Poetry version: <1.0.0>

Issue

I ran into an error that pops up as: expected string or bytes-like object. I followed the stack trace and found the location where it happens: /.poetry/lib/poetry/repositories/installed_repository.py

The problem is in the load() function where it calls to get a list of paths or something using the env.sys_path variable. For me it looked like this:

'''
for distribution in sorted(
metadata.distributions(path=env.sys_path), key=lambda d: str(d._path),
):
'''

But I know that this is a little different between versions, so I'll specifically say that the issue was the env.sys_path. My env.sys_path was returning one string that's empty in it's list. I don't know why, I just know it does that. The empty string then is passed along with the rest of the code which assumes that it is a distribution package with metadata. It specifically passes along a None value which then breaks things later. The load() function needs to have logic to handle the case where env.sys_path has an empty string in the return values.

I was able to fix it by adding a simple check after the values are returned:

            name = distribution.metadata["name"]
            if name is None:
                continue

I tried cleaning the names for env.sys_path before they're used but then I get a different weird error:

[TypeError]
find_distributions() got an unexpected keyword argument 'name'

So I just did the simple fix where I check if name is None.

Simple fix, should be quick to add to the next release.

Stack trace:

[TypeError]
expected string or bytes-like object

Traceback (most recent call last):
  File "/home/.poetry/lib/poetry/_vendor/py3.6/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/home/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/.poetry/lib/poetry/console/commands/install.py", line 48, in handle
    self.io, self.env, self.poetry.package, self.poetry.locker, self.poetry.pool
  File "/home/.poetry/lib/poetry/installation/installer.py", line 55, in __init__
    installed = self._get_installed()
  File "/home/.poetry/lib/poetry/installation/installer.py", line 488, in _get_installed
    return InstalledRepository.load(self._env)
  File "/home/.poetry/lib/poetry/repositories/installed_repository.py", line 24, in load
    package = Package(name, version, version)
  File "/home/.poetry/lib/poetry/packages/package.py", line 42, in __init__
    self._name = canonicalize_name(name)
  File "/home/.poetry/lib/poetry/utils/helpers.py", line 31, in canonicalize_name
    return _canonicalize_regex.sub('-', name).lower()

[ezdac]

I get the same error and stack trace, although I can't comment right now on what the env.sys_path looks like for me.
I'm running this with an active miniconda environment, set by pyenv - the error occurs on poetry install.

Mac OSX 10.14.5
poetry 1.0.3

[languitar]

Any idea how to work around this issue? I am hit by this as well right now in a CI build but it doesn't happen on my local machine with the same poetry version being used.

[languitar]

I debugged this a bit further. For me this occurs when VIRTUAL_ENV is set and the env.sys_path call redirect to the virtual environment. In that environment, the following happens:

>>> import sys
>>> sys.path
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/tmp/foo/.tox/check/lib/python3.8/site-packages']
>>> os.environ
environ({'SHELL': '/bin/bash', 'TMUX': '/tmp//tmux-0/default,588,0', 'HOSTNAME': 'fdc08107af3e', 'PYTHON_VERSION': '3.8.1', 'PWD': '/tmp/foo', 'HOME': '/root', 'LANG': 'C.UTF-8', 'GPG_KEY': 'E3FF2839C048B25C084DEBE9B26995E310250568', 'TERM': 'screen', 'TMUX_PANE': '%1', 'SHLVL': '2', 'PYTHON_PIP_VERSION': '20.0.2', 'PYTHON_GET_PIP_SHA256': '421ac1d44c0cf9730a088e337867d974b91bdce4ea2636099275071878cc189e', 'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/d59197a3c169cef378a22428a3fa99d33e080a5d/get-pip.py', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'OLDPWD': '/tmp', '_': '/tmp/foo/.tox/check/bin/python'})

[languitar]

Some further reference on why this happens: https://docs.python.org/3/library/sys.html#sys.path

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.

[sisp]

I'm experiencing the same issue when using tox. pip install tox now installs virtualenv==20.0.0, but a few days ago 16.7.9 was installed. Forcing virtualenv<20, i.e. pip install 'virtualenv<20' tox, solves the problem for me, but it's only a quick fix.

[gjedlicska]

I can confirm the workaround by @sisp. For me the issue came up, when I added a package dependency from a private package index and tried to run tests with tox. Running poetry install with our without --no-root fails with the same symptoms.

[gjedlicska]

I did some more digging around, it looks like the package metadata from virtualenv 20.0 is missing

{'policy': Compat32(), '_headers': [], '_unixfrom': None, '_payload': '', '_charset': None, 'preamble': None, 'epilogue': None, 'defects': [], '_default_type': 'text/plain'}
in the headers section all other packages contain a (Name: "foo") and (Version: bar).

Should this be handled inside poetry, or should this be raised as a bug to virtualenv?
I'm not an expert on this yet.

[alalazo]

pip install tox now installs virtualenv==20.0.0, but a few days ago 16.7.9 was installed.

+1 to the number of people hit by this

[peterdeme]

Tox does not pin down the virtualenv version, so this seems broken (for everyone?) since 10 February.

https://github.com/tox-dev/tox/blob/6472eac8fc4f0c222d9f40c4572aafbca72c40db/setup.cfg#L47

[cjolowicz]

Here is a reproduction using Docker:

https://gist.github.com/cjolowicz/078edda96f44ab593cc6442ac9ebaa8d

Run like this:

git clone https://gist.github.com/078edda96f44ab593cc6442ac9ebaa8d.git
cd 078edda96f44ab593cc6442ac9ebaa8d/
sh test.sh

[cjolowicz]

This issue is not caused by an empty string in env.sys_path, as the title says. (As mentioned above, the empty string is standard behaviour. Also, the exception is not raised on the code path involving the empty string, but when processing the site-packages directory.)

Rather, the virtual environments created by virtualenv >= 20.0 contain marker files, which cause importlib.metadata.distributions() to report duplicate entries for packages, with None as their name and version.

This has been reported to virtualenv here: pypa/virtualenv#1589

[cjolowicz]

The fix is in virtualenv's master.

On a related note, Poetry depends on a rather old version of importlib_metadata (~1.1.3) for Python pre-3.8. Newer versions of this library are not affected by the bug. Upgrading to importlib_metadata 1.4 or 1.5 would prevent Poetry from being affected by this bug.

Python 3.8.2 (due 2020-02-17) incorporates changes from importlib_metadata 1.4 and is therefore also not affected by the bug.

[languitar]

What of the different options is likely to result in the fastest fix for users? Would be nice to get this resolved quickly. So far, I have to patch all CI pipelines I am maintaining to avoid the issue.

[cjolowicz]

This is resolved with virtualenv 20.0.3, released today.

@languitar Upgrade your CI to the latest virtualenv by removing the pin or pointing it to 20.0.3.

[finswimmer]

Thank you all for reporting and keeping the community up-to-date.

Good work 👍

[classic-paul]

I'm still seeing this whilst using nox:

def deps(session):
    session.install("virtualenv<20", "poetry")
    session.run("poetry", "install")

results in [TypeError] expected string or bytes-like object

Same result for virtualenv versions 20.0.3 and 20.0.4.
Can't be sure if this is specific to nox (and it's especially odd as in this instance on a Windows box it's behaving fine, but on a Linux box... nope).

[cjolowicz]

@pcp1976 Can you try deleting virtualenv's user data directory, as described here?

[classic-paul]

Yep @cjolowicz, that's fixed it - lovely job, thanks.

[uelei]

So I also having the same problem, and i dont have virtualenv installed, only pyenv + pipx + poetry

argcomplete==1.11.1
Click==7.0
distro==1.4.0
importlib-metadata==1.4.0
pipx==0.15.1.3
userpath==1.3.0
zipp==3.0.0

when i debug poetry I'm having a empty string as sys.path

['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/uelei/.cache/pypoetry/virtualenvs/project-kQO-PhBL-py3.6/lib/python3.6/site-packages']

and that first "" was breaking poetry

so I made this PR to fix it #2092

[revmischa]

I deleted my virtualenv user data and ran pip3 install -U virtualenv to upgrade to latest virtualenv but am still seeing this issue.
Python 3.8.1

[cjolowicz]

@uelei The empty string in sys.path is Python's way of representing the current directory.

@uelei and @revmischa: Can you provide some more information about the problem you encounter (error message, traceback if possible), which environment you run in (operating system, versions of Python and relevant tools), and ideally how to reproduce the issue?

The problem identified earlier is caused by marker files in the virtual environment, with the file extension .dist-info.virtualenv. Can you check if you see any of these files in the virtual environment used by Poetry when you encounter the error?

Thanks for your reports!

[stefsmeets]

I deleted my virtualenv user data and ran pip3 install -U virtualenv to upgrade to latest virtualenv but am still seeing this issue.
Python 3.8.1

I'm running into the same issue as described by OP after upgrading some packages. Tried upgrading/downgrading virtualenv, removing appdata, and the other suggestions described here. Not sure how to proceed. This is on Python 3.8.2 on Windows 10 (poetry v1.0.5, virtualenv v20.0.20).

poetry install -vvv

[TypeError] expected string or bytes-like object

Traceback (most recent call last):
  File "C:\Users\Stef\.poetry\lib\poetry\_vendor\py3.8\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\Stef\.poetry\lib\poetry\_vendor\py3.8\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\Stef\.poetry\lib\poetry\_vendor\py3.8\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\Stef\.poetry\lib\poetry\_vendor\py3.8\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\Stef\.poetry\lib\poetry\console\commands\install.py", line 47, in handle
    installer = Installer(
  File "C:\Users\Stef\.poetry\lib\poetry\installation\installer.py", line 55, in __init__
    installed = self._get_installed()
  File "C:\Users\Stef\.poetry\lib\poetry\installation\installer.py", line 488, in _get_installed
    return InstalledRepository.load(self._env)
  File "C:\Users\Stef\.poetry\lib\poetry\repositories\installed_repository.py", line 28, in load
    package = Package(name, version, version)
  File "C:\Users\Stef\.poetry\lib\poetry\packages\package.py", line 43, in __init__
    self._name = canonicalize_name(name)
  File "C:\Users\Stef\.poetry\lib\poetry\utils\helpers.py", line 26, in canonicalize_name
    return _canonicalize_regex.sub("-", name).lower()

[stefsmeets]

A simple work-around is to change https://github.com/python-poetry/poetry/blob/master/poetry/repositories/installed_repository.py#L25 to:

                name = distribution.metadata["name"]
                if not name:
                    continue

[absassi]

To those still having problems even after upgrading virtualenv, see this comment (in short, remove any .egg-info folder from your project).

[sisp]

@stefsmeets When you create a new empty virtual environment (with Virtualenv or Conda) and run pip list inside the virtual environment, only a few basic packages should be listed. What do you see?

I've encountered this problem when packages from other site-packages directories leak into the virtual environment, i.e. when other site-packages directories are in the search path.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.