typing breaks local package installs

[awilkins]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Ubuntu 20.04:
• Poetry 1.1.0:

Issue

Installing typing as a direct or transitive dependency breaks pip in Python versions 3.5+ as per this issue.

This seems to break installing local packages from relative paths in poetry 1.1.0 but not 1.0.10

Looking at the stack trace I can see that 1.1.0 is running pip out of the project virtualenv, not it's own virtualenv. As I understood it, Poetry "vendorizes it's dependencies" to avoid this, which doesn't seem consistent with running pip from a virtualenv you are managing.

Reproduce

# Python 3.7.9
❯ poetry new submodule
❯ poetry new test
❯ cd test
❯ poetry add typing
❯ poetry add ../submodule -vvv
Using virtualenv: /home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7

Updating dependencies
Resolving dependencies...
   1: fact: test2 is 0.1.0
   1: derived: test2
   1: fact: test2 depends on typing (^3.7.4)
   1: fact: test2 depends on submodule (0.1.0 ../submodule)
   1: fact: test2 depends on pytest (^5.2)
   1: fact: test2 depends on pytest (^5.2)
   1: selecting test2 (0.1.0)
   1: derived: pytest (^5.2)
   1: derived: submodule (0.1.0 ../submodule)
   1: derived: typing (^3.7.4)
   1: fact: pytest (5.4.3) depends on py (>=1.5.0)
   1: fact: pytest (5.4.3) depends on packaging (*)
   1: fact: pytest (5.4.3) depends on attrs (>=17.4.0)
   1: fact: pytest (5.4.3) depends on more-itertools (>=4.0.0)
   1: fact: pytest (5.4.3) depends on pluggy (>=0.12,<1.0)
   1: fact: pytest (5.4.3) depends on wcwidth (*)
   1: fact: pytest (5.4.3) depends on importlib-metadata (>=0.12)
   1: fact: pytest (5.4.3) depends on atomicwrites (>=1.0)
   1: fact: pytest (5.4.3) depends on colorama (*)
   1: selecting pytest (5.4.3)
   1: derived: colorama (*)
   1: derived: atomicwrites (>=1.0)
   1: derived: importlib-metadata (>=0.12)
   1: derived: wcwidth (*)
   1: derived: pluggy (>=0.12,<1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: packaging (*)
   1: derived: py (>=1.5.0)
   1: selecting submodule (0.1.0 /home/awilkins/poetry/submodule)
   1: selecting typing (3.7.4.3)
   1: selecting colorama (0.4.3)
   1: selecting atomicwrites (1.4.0)
   1: fact: importlib-metadata (2.0.0) depends on zipp (>=0.5)
   1: selecting importlib-metadata (2.0.0)
   1: derived: zipp (>=0.5)
   1: selecting wcwidth (0.2.5)
   1: fact: pluggy (0.13.1) depends on importlib-metadata (>=0.12)
   1: selecting pluggy (0.13.1)
   1: selecting more-itertools (8.5.0)
   1: selecting attrs (20.2.0)
   1: fact: packaging (20.4) depends on pyparsing (>=2.0.2)
   1: fact: packaging (20.4) depends on six (*)
   1: selecting packaging (20.4)
   1: derived: six (*)
   1: derived: pyparsing (>=2.0.2)
   1: selecting py (1.9.0)
   1: selecting zipp (3.2.0)
   1: selecting six (1.15.0)
   1: selecting pyparsing (2.4.7)
   1: Version solving took 0.085 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 14 skipped

  • Removing atomicwrites (1.4.0): Pending...
  • Removing atomicwrites (1.4.0): Skipped for the following reason: Not currently installed
  • Removing colorama (0.4.3): Pending...
  • Removing colorama (0.4.3): Skipped for the following reason: Not currently installed
  • Installing zipp (3.2.0): Pending...
  • Installing zipp (3.2.0): Skipped for the following reason: Already installed
  • Installing importlib-metadata (2.0.0): Pending...
  • Installing importlib-metadata (2.0.0): Skipped for the following reason: Already installed
  • Installing pyparsing (2.4.7): Pending...
  • Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
  • Installing six (1.15.0): Pending...
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing attrs (20.2.0): Pending...
  • Installing attrs (20.2.0): Skipped for the following reason: Already installed
  • Installing more-itertools (8.5.0): Pending...
  • Installing more-itertools (8.5.0): Skipped for the following reason: Already installed
  • Installing packaging (20.4): Pending...
  • Installing packaging (20.4): Skipped for the following reason: Already installed
  • Installing pluggy (0.13.1): Pending...
  • Installing pluggy (0.13.1): Skipped for the following reason: Already installed
  • Installing py (1.9.0): Pending...
  • Installing py (1.9.0): Skipped for the following reason: Already installed
  • Installing wcwidth (0.2.5): Pending...
  • Installing wcwidth (0.2.5): Skipped for the following reason: Already installed
  • Installing pytest (5.4.3): Pending...
  • Installing pytest (5.4.3): Skipped for the following reason: Already installed
  • Installing submodule (0.1.0 /home/awilkins/poetry/submodule): Pending...
  • Installing submodule (0.1.0 /home/awilkins/poetry/submodule): Building...
  • Installing submodule (0.1.0 /home/awilkins/poetry/submodule): Failed

  EnvCommandError

  Command ['/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/bin/pip', 'install', '--no-deps', '-U', '/home/awilkins/poetry/submodule'] errored with the following return code 1, and output: 
  Processing /home/awilkins/poetry/submodule
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/bin/python /home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5p7x258v/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry-core>=1.0.0'
         cwd: None
    Complete output (42 lines):
    Traceback (most recent call last):
      File "/home/awilkins/.asdf/installs/python/3.7.9/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/awilkins/.asdf/installs/python/3.7.9/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/__main__.py", line 26, in <module>
        sys.exit(_main())
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 73, in main
        command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
        module = importlib.import_module(module_path)
      File "/home/awilkins/.asdf/installs/python/3.7.9/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 17, in <module>
        from pip._internal.cli.req_command import RequirementCommand, with_cleanup
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
        from pip._internal.index.collector import LinkCollector
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 14, in <module>
        from pip._vendor import html5lib, requests
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_vendor/requests/__init__.py", line 125, in <module>
        from . import utils
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
        from . import certs
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
        from pip._vendor.certifi import where
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
        from .core import contents, where
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
        from importlib.resources import path as get_path, read_text
      File "/home/awilkins/.asdf/installs/python/3.7.9/lib/python3.7/importlib/resources.py", line 11, in <module>
        from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/typing.py", line 1359, in <module>
        class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
      File "/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/typing.py", line 1007, in __new__
        self._abc_registry = extra._abc_registry
    AttributeError: type object 'Callable' has no attribute '_abc_registry'
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/bin/python /home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5p7x258v/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry-core>=1.0.0' Check the logs for full command output.
  WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
  You should consider upgrading via the '/home/awilkins/.cache/pypoetry/virtualenvs/test2-fL2i9h4Q-py3.7/bin/python -m pip install --upgrade pip' command.
  

  at ~/.poetry/lib/poetry/utils/env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│ 
       950│         return decode(output)
       951│ 
       952│     def execute(self, bin, *args, **kwargs):

  • Installing typing (3.7.4.3): Pending...
  • Installing typing (3.7.4.3): Skipped for the following reason: Already installed

Failed to add packages, reverting the pyproject.toml file to its original content.

❯ poetry self update 1.0.10
Updating to 1.0.10
 - Downloading poetry-1.0.10-linux.tar.gz 100%

Poetry (1.0.10) is installed now. Great!

~/poetry/test2 is 📦 v0.1.0 via 🐍 v3.7.9 on ☁️  eu-west-1 took 9s 
❯ poetry add ../submodule/

The lock file might not be compatible with the current version of Poetry.
Upgrade Poetry to ensure the lock file is read properly or, alternatively, regenerate the lock file with the `poetry lock` command.
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing submodule (0.1.0 ../submodule)

[abn]

@awilkins please refer to https://python-poetry.org/blog/announcing-poetry-1-1-0.html#directory-dependencies-are-now-non-editable-by-default

Seems pip relies on a particular version of typing when doing a PEP 517 build of a path dependency, hence why the failure.

[awilkins]

Seems pip relies on a particular version of typing when doing a PEP 517 build of a path dependency

They seem to acknowledge that even doing this is a bug in this one

Adding { develop = true } does fix my problem, I'm building Lambda packages and layers and my packaging script already de-references egg-links

[abn]

@awilkins closing this as it seems this might not be a poetry issue as things stand.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.