Pipenv installs spurious dependencies

[J-M0]

Issue description

When installing any version of Django < 3.0 as a dependency and django-debug-toolbar as a dev dependency, pipenv lock determines that asgiref is a required package when it in fact is not.

Expected result

$ pipenv graph
django-debug-toolbar==2.2
  - Django [required: >=1.11, installed: 2.2.13]
    - pytz [required: Any, installed: 2020.1]
    - sqlparse [required: Any, installed: 0.3.1]
  - sqlparse [required: >=0.2.0, installed: 0.3.1]

Actual result

$ pipenv graph
asgiref==3.2.10
django-debug-toolbar==2.2
  - Django [required: >=1.11, installed: 2.2.13]
    - pytz [required: Any, installed: 2020.1]
    - sqlparse [required: Any, installed: 0.3.1]
  - sqlparse [required: >=0.2.0, installed: 0.3.1]

Steps to replicate

In a new project, run the following commands:

$ pipenv install 'Django<3.0'
$ pipenv install --dev django-debug-toolbar

---

$ pipenv --support

Pipenv version: '2020.6.2'

Pipenv location: '/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2020.6.2/libexec/bin/python3.8'

Python installations found:

• 3.8.0: /Users/jmorris/.pyenv/versions/3.8.0/bin/python3
• 3.7.7: /usr/local/bin/python3
• 3.7.7: /usr/local/bin/python3.7m
• 3.7.7: /usr/local/bin/python3.7
• 3.6.8: /Users/jmorris/.pyenv/versions/3.6.8/bin/python3
• 3.5.9: /Users/jmorris/.pyenv/versions/3.5.9/bin/python3
• 3.5.1: /Users/jmorris/.pyenv/versions/3.5.1/bin/python3
• 2.7.18: /Users/jmorris/.pyenv/versions/2.7.18/bin/python2
• 2.7.17: /usr/local/bin/python2
• 2.7.17: /usr/local/bin/python2.7
• 2.7.16: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT '
                     '2020; root:xnu-4903.278.35~1/RELEASE_X86_64',
 'python_full_version': '3.8.3',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• TERM_PROGRAM
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• USER
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PWD
• EDITOR
• LANG
• ITERM_PROFILE
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• PYENV_SHELL
• HOME
• COLORFGBG
• LC_TERMINAL_VERSION
• ITERM_SESSION_ID
• LOGNAME
• LESS
• LESSOPEN
• LC_TERMINAL
• COLORTERM
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2020.6.2/libexec/tools:/Users/jmorris/.pyenv/shims:/Users/jmorris/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/MacGPG2/bin:/Applications/Wireshark.app/Contents/MacOS
• SHELL: /usr/local/bin/fish
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /tmp/lol

---

Contents of Pipfile ('/private/tmp/lol/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
django-debug-toolbar = "*"

[packages]
django = "<3.0"

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/private/tmp/lol/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "5e37dff5e53a743305538fdd5fb9da0dbe7eea0521b520e50a56ee191050c830"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django": {
            "hashes": [
                "sha256:84f370f6acedbe1f3c41e1a02de44ac206efda3355e427139ecb785b5f596d80",
                "sha256:e8fe3c2b2212dce6126becab7a693157f1a441a07b62ec994c046c76af5bb66d"
            ],
            "index": "pypi",
            "version": "==2.2.13"
        },
        "pytz": {
            "hashes": [
                "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
                "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
            ],
            "version": "==2020.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e",
                "sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==0.3.1"
        }
    },
    "develop": {
        "asgiref": {
            "hashes": [
                "sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a",
                "sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.2.10"
        },
        "django": {
            "hashes": [
                "sha256:84f370f6acedbe1f3c41e1a02de44ac206efda3355e427139ecb785b5f596d80",
                "sha256:e8fe3c2b2212dce6126becab7a693157f1a441a07b62ec994c046c76af5bb66d"
            ],
            "index": "pypi",
            "version": "==2.2.13"
        },
        "django-debug-toolbar": {
            "hashes": [
                "sha256:eabbefe89881bbe4ca7c980ff102e3c35c8e8ad6eb725041f538988f2f39a943",
                "sha256:ff94725e7aae74b133d0599b9bf89bd4eb8f5d2c964106e61d11750228c8774c"
            ],
            "index": "pypi",
            "version": "==2.2"
        },
        "pytz": {
            "hashes": [
                "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
                "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
            ],
            "version": "==2020.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e",
                "sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==0.3.1"
        }
    }
}

[stianjensen]

Not sure why it ends up at the top-level, but django does require asgiref in setup.cfg:
https://github.com/django/django/blob/197b55c53469cf8344d1ba35175236780cb83bd1/setup.cfg#L43-L46

[J-M0]

Yes, Django started requiring asgiref in Django 3. If you look at the requirements for Django 2.2.x, asgiref is absent:

https://github.com/django/django/blob/stable/2.2.x/setup.py#L86

[GPHemsley-RELX]

The linked #4455 provides more context: The packages that are being installed are future dependencies; that is, dependencies of later versions of packages that --keep-outdated is, well, keeping outdated. They shouldn't even enter into the dependency tree if their parent package is being held back.

This is still a problem in pipenv 2022.1.8.

[matteius]

@GPHemsley-RELX Can you check if it is still a problem in my branch that upgrades to latest pip? Branch is vendor-pip-22.0.3-followup-changes and the new pip resolver solves some other infinity lock issues so I am wondering if the resolver fixes that as well.

EDIT: Nevermind, I just checked the steps it appears to still be a problem even on my branch:

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4371$ pipenv graph
asgiref==3.5.0
django-debug-toolbar==3.2.4
  - Django [required: >=2.2, installed: 2.2.27]
    - pytz [required: Any, installed: 2021.3]
    - sqlparse [required: >=0.2.2, installed: 0.4.2]
  - sqlparse [required: >=0.2.0, installed: 0.4.2]

Just noting this is specifically an issue of installing a new dependency as --dev because if I repeat the steps but install pipenv install django-debug-toolbar regularly it produces the expected graph.

[dqkqd]

@matteius My current solution is creating default constraints associated with packages, then pass those constraints to resolver and use them when installing dev-packages. But it only solves the case when installing from Pipfile, not the case where user provides packages.

# Pipfile
[packages]
django = "<3.0"

[dev-packages]
django-debug-toolbar = "*"

$ pipenv graph
django-debug-toolbar==3.2.4
  - Django [required: >=2.2, installed: 2.2.28]
    - pytz [required: Any, installed: 2022.1]
    - sqlparse [required: >=0.2.2, installed: 0.4.2]
  - sqlparse [required: >=0.2.0, installed: 0.4.2]

Should I open a PR just for this case?

[matteius]

New version of pipenv==2022.8.13 has been released with the fix produced by @dqkqd !