Locking does not respect editable flag for local packages in multiple categories

[micahjsmith]

Issue description

If a local package is specified in both default and develop categories with different values for editable, after locking the value from the default category is used in both places in the lockfile. If the local package is not provided in default category, then the locking works correctly in that the value of editable is correct in the lockfile's develop category. I haven't found any specification that says that the same package can't be provided in multiple categories, so I'd expect this to work properly.

As context, the use case for having the local package specified twice is that at deployment time you want to just build a wheel and install that without any symlinks etc., whereas at development time, you want to have the editable distribution.

Expected result

in the lockfile, editable is false for default category and true for develop category

    "default": {
        "packaging": {
            "path": "./packaging",
            "version": "==21.4.dev0"
        }
    },
    "develop": {
        "packaging": {
            "path": "./packaging",
            "version": "==21.4.dev0",
            "editable": true
        }
    }

Actual result

in the lockfile, editable is false for default category and false for develop category

    "default": {
        "packaging": {
            "path": "./packaging",
            "version": "==21.4.dev0"
        }
    },
    "develop": {
        "packaging": {
            "path": "./packaging",
            "version": "==21.4.dev0"
        }
    }

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

mkdir pipenv_sandbox
cd pipenv_sandbox
git clone https://github.com/pypa/packaging
# populate Pipfile as below in details section
pipenv lock --dev
# observe that editable is set to False in develop category

---

$ pipenv --support

Pipenv version: '2022.10.12'

Pipenv location: '/Users/micahsmith/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pipenv'

Python location: '/Users/micahsmith/.pyenv/versions/3.9.13/bin/python3.9'

OS Name: 'posix'

User pip version: '22.2.2'

user Python installations found:

• 3.10.7: /usr/local/bin/python3
• 3.10.6: /Users/micahsmith/.pyenv/versions/3.10.6/bin/python3
• 3.9.15: /usr/local/bin/python3.9
• 3.9.13: /Users/micahsmith/.pyenv/versions/3.9.13/bin/python3
• 3.9.13: /Users/micahsmith/.pyenv/versions/3.9.13/bin/python3
• 3.9.13: /Users/micahsmith/.pyenv/versions/3.9.13/bin/python
• 3.9.13: /Users/micahsmith/.pyenv/versions/3.9.13/bin/python3.9
• 3.9.6: /usr/bin/python3
• 3.8.15: /usr/local/bin/python3.8
• 3.8.3: /Users/micahsmith/.pyenv/versions/3.8.3/bin/python3
• 3.7.7: /Users/micahsmith/.pyenv/versions/3.7.7/bin/python3
• 3.6.13: /Users/micahsmith/.pyenv/versions/3.6.13/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.13',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '21.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT '
                     '2022; root:xnu-8020.140.49~2/RELEASE_X86_64',
 'python_full_version': '3.9.13',
 'python_version': '3.9',
 'sys_platform': 'darwin'}

System environment variables (redacted):

• PYENV_ROOT
• PIPENV_VENV_IN_PROJECT
• PYENV_VERSION
• PIPENV_IGNORE_VIRTUALENVS
• PYENV_DIR
• PYENV_VIRTUALENV_INIT
• PYENV_HOOK_PATH
• PYENV_SHELL
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1
• PIPENV_IGNORE_VIRTUALENVS: ``

Debug–specific environment variables:

• PATH: /Users/micahsmith/.pyenv/versions/3.9.13/bin:/usr/local/Cellar/pyenv/2.3.5/libexec:/usr/local/Cellar/pyenv/2.3.5/plugins/python-build/bin:/usr/local/Cellar/pyenv-virtualenv/1.1.5/shims:/Users/micahsmith/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/micahsmith/.gem/ruby/2.7.0/bin:/usr/local/opt/ruby/bin:/usr/local/opt/libxml2/bin:/Users/micahsmith/go/bin:/usr/local/sbin:/usr/local/opt/util-linux/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/Cellar/pyenv-virtualenv/1.1.5/shims:/Users/micahsmith/.local/bin:/Users/micahsmith/local/bin:/usr/local/opt/fzf/bin
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /Users/micahsmith/workspace/pipenv_sandbox

---

Contents of Pipfile ('/Users/micahsmith/workspace/pipenv_sandbox/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
packaging = {path = "./packaging", editable = false}

[dev-packages]
packaging = {path = "./packaging", editable = true}

[requires]
python_version = "3.10"
python_full_version = "3.10.7"

Contents of Pipfile.lock ('/Users/micahsmith/workspace/pipenv_sandbox/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "acd2f70bd2a5c1fbe0c7506327757fc7e6638811524980dc6c5f02376f91b47d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.10.7",
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "packaging": {
            "editable": false,
            "path": "./packaging",
            "version": "==21.4.dev0"
        }
    },
    "develop": {
        "packaging": {
            "editable": false,
            "path": "./packaging",
            "version": "==21.4.dev0"
        }
    }
}

[oz123]

Thank you for the thorough bug report. We'll try to fix this as soon as possible.
We have full time jobs, so it might take a while!

[matteius]

@micahjsmith and @oz123 -- Actually this is intentional. The default package group constraints all other categories. This is also called out in the documentation: https://pipenv.pypa.io/en/latest/basics/#specifying-package-categories

Note
The packages/default specifiers are used to constrain all other categories just as they have done for dev-packages/develop category. However this is the only way constraints are applied – the presence of other named groups do not constraint each other, which means it is possible to define conflicting package versions across groups. This may be desired in some use cases where users only are installing groups specific to their system platform.

The original fix to constraint dev dependencies with default category came from @dqkqd which resolved a number of other issue reports in the backlog. See original PR and trace back of issues for more details: #5234

[matteius]

One possible user workaround if you have packages you don't want constrained by default, or if you don't like that behavior, you can opt to use different categories and potentially even leave default empty, and nothing will be constrained across categories -- it would be a bit like the wild west to not care about conflicting versions; it would be on the user to understand which groups are compatible/incompatible, but I think that is the best we can do to continue to support both use cases.

[micahjsmith]

Hi @matteius thanks for the explanation.

I had missed this section, but even after reading it, it's not fully clear what it means

The packages/default specifiers are used to constrain all other categories just as they have done for dev-packages/develop category.

After reading through the linked PR, I understand that this means "constrain" in the since of "pip constraints". My takeaway is that if the same dependency is specified in both packages and category X, the version in packages section "wins" during resolution.

Regarding the order in which package categories are resolved:

The command should process the package groups in the order specified.

It seems that this has the important caveat that the packages group (default) is always processed last, yes?

[micahjsmith]

Thanks for the idea for the workaround

One possible user workaround if you have packages you don't want constrained by default, or if you don't like that behavior, you can opt to use different categories and potentially even leave default empty, and nothing will be constrained across categories

Here was my use case from above

As context, the use case for having the local package specified twice is that at deployment time you want to just build a wheel and install that without any symlinks etc., whereas at development time, you want to have the editable distribution.

Since the local package is required in the environment, the proposed workaround would mean that one could not create a satisfactory environment that has all required packages just by doing pipenv sync, one would install always have to do pipenv sync --categories "default editable" (modulo syntax on how to use categories flag). This feels like a major degradation in usability of pipenv as it pertains to this use case.

For now, I am considering a workaround of setting the local package as always editable, then uninstalling/reinstalling the package in a post-processing step when I need a non-editable install such as a in deployment context

pipenv sync
python -m pip uninstall -y local-package \
    && python -m pip install --no-deps ./local-package

[matteius]

@micahjsmith Thanks for your feedback; I will contemplate it more. I think the original issue stemmed from say I have Django in default and django-debug-toolbar in dev -- I don't want that dev dependency to pull in a newer version of Django than what I am developing against. Perhaps an option of adding to the pipenv section of the Pipfile to not constrain other categories by the default category could be an option, again would need to think more about it.

For your workaround idea, that is always an option, but you may want to use pipenv to run pip install (in the case of virtualenvs or to ensure you are using the same python version):

pipenv sync
pipenv run pip uninstall -y local-package \
    && pipenv run pip install --no-deps ./local-package

[micahjsmith]

Makes sense, thanks for the discussion, going to close this then as expected behavior