Dependency erroneously being marked local and editable in Pipfile.lock

[jillianrosile]

Issue description

I believe this is very similar to this closed issue but due to a .serverless/requirements directory for serverless rather than the venv directory in that issue. It appears to be an issue with any subdirectory that contains .egg-info folders, but I'm providing the steps to reproduce for the serverless-specific example.

The last external package listed in the default section of Pipfile.lock is being marked as local and editable when an editable self-dependency exists. This happens when running pipenv lock after running a serverless command that creates the .serverless/requirements directory.

I'm running pipenv version 2020.11.15.

Expected result

Last external package is provided from pypi.

Actual result

Last external package is marked editable and local, example:

        "pytz": {
            "editable": true,
            "path": "."
        }

Steps to replicate

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
pytz = "*"
my-project = { editable = true, path = "." }

[requires]
python_version = "3.8"

setup.cfg:

[metadata]
name = my-project
description = Test project to reproduce issue

[options]
zip_safe = False
include_package_data = True
package_dir =
    =src/
packages = find_namespace:
python_requires = >=3.8,<3.9

[options.packages.find]
where = src/

setup.py:

from setuptools import setup
setup()

You'll need a src folder with __init__.py, too.

pipenv lock runs as expected at this point with no bug when there is no .serverless/requirements directory.

Add a basic serverless.yml file to the project root, e.g.

service: my-project

provider:
  name: aws
  runtime: python3.8
  timeout: 30
  stage: ${opt:stage, 'alpha'}

package:
  include:
    - ./src/**
  exclude:
    - ./**

plugins:
  - serverless-python-requirements

Then run sls plugin install -n serverless-python-requirements and then sls package. This command will fail with the example setup, which doesn't matter, this bug will still be triggered.

Run pipenv lock again.

Diff:

@@ -21,12 +21,8 @@
             "path": "."
         },
         "pytz": {
-            "hashes": [
-                "sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da",
-                "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798"
-            ],
-            "index": "pypi",
-            "version": "==2021.1"
+            "editable": true,
+            "path": "."
         }
     },
     "develop": {}

---

The output of pipenv --support contained some things I'm not comfortable posting publicly related to my setup / environment vars, but if there are any snippets of it that aren't already included above that you need to reproduce this, please let me know and I can provide those.

[matteius]

@jillianrosile Other than the issue of having to do the editable install twice because the first time it gets a Text File Busy error, I find that I do not get the same bug as you describe using latest version pipenv==2021.11.23

mdavis@matt-VirtualBox:~/shared-projects/pipenv-4686$ pipenv lock
Creating a virtualenv for this project...
Pipfile: /home/mdavis/shared-projects/pipenv-4686/Pipfile
Using /home/mdavis/.pyenv/versions/3.8.12/bin/python3.8 (3.8.12) to create virtualenv...
⠦ Creating virtual environment...created virtual environment CPython3.8.12.final.0-64 in 295ms
  creator CPython3Posix(dest=/home/mdavis/.virtualenvs/pipenv-4686-ma8Gs5CE, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mdavis/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==59.2.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/mdavis/.virtualenvs/pipenv-4686-ma8Gs5CE
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (82474d)!
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/mdavis/.local/lib/python3.9/site-packages/pipenv/vendor/vistir/path.py", line 433, in rmtree
    shutil.rmtree(directory, ignore_errors=ignore_errors, onerror=onerror)
  File "/usr/lib/python3.9/shutil.py", line 726, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/mdavis/.local/lib/python3.9/site-packages/pipenv/vendor/vistir/path.py", line 528, in handle_remove_readonly
    raise exc_exception
  File "/usr/lib/python3.9/shutil.py", line 665, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 26] Text file busy: 'my_project.egg-info'
mdavis@matt-VirtualBox:~/shared-projects/pipenv-4686$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (82474d)!
mdavis@matt-VirtualBox:~/shared-projects/pipenv-4686$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "21adc73cb451fda3e95defb73cca6b393116b3f430bcc299de9ca97c4d82474d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "my-project": {
            "editable": true,
            "path": "."
        },
        "pytz": {
            "hashes": [
                "sha256:3672058bc3453457b622aab7a1c3bfd5ab0bdae451512f6cf25f64ed37f5b87c",
                "sha256:acad2d8b20a1af07d4e4c9d2e9285c5ed9104354062f275f3fcd88dcef4f1326"
            ],
            "index": "pypi",
            "version": "==2021.3"
        }
    },
    "develop": {}
}
mdavis@matt-VirtualBox:~/shared-projects/pipenv-4686$ pipenv --version
pipenv, version 2021.11.23

EDIT: Actually the initial error had to do with my shared VM filesystem, so I will try this again now.

[matteius]

@jillianrosile I just tried it out with 2022.1.8 and did not experience the issue.

Running "serverless" from node_modules

✔ Plugin "serverless-python-requirements" installed  (2s)
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ sls package
Running "serverless" from node_modules

Packaging my-project for stage alpha (us-east-1)
Environment: linux, node 14.19.0, framework 3.7.4 (local) 3.7.4v (global), plugin 6.1.5, SDK 4.3.2
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Error: spawn python3.8 ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:274:19)
    at onErrorNT (internal/child_process.js:469:16)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)

1 deprecation found: run 'serverless doctor' for more details
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ ls
build  node_modules  package.json  package-lock.json  Pipfile  Pipfile.lock  serverless.yml  setup.cfg  setup.py  src
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "21adc73cb451fda3e95defb73cca6b393116b3f430bcc299de9ca97c4d82474d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "my-project": {
            "editable": true,
            "path": "."
        },
        "pytz": {
            "hashes": [
                "sha256:3672058bc3453457b622aab7a1c3bfd5ab0bdae451512f6cf25f64ed37f5b87c",
                "sha256:acad2d8b20a1af07d4e4c9d2e9285c5ed9104354062f275f3fcd88dcef4f1326"
            ],
            "index": "pypi",
            "version": "==2021.3"
        },
        "wrapt": {
            "editable": true,
            "path": "."
        }
    },
    "develop": {}
}
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ pipen^C
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ cp Pipfile.lock Pipfile.lock.initial
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (82474d)!
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "21adc73cb451fda3e95defb73cca6b393116b3f430bcc299de9ca97c4d82474d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "my-project": {
            "editable": true,
            "path": "."
        },
        "pytz": {
            "hashes": [
                "sha256:3672058bc3453457b622aab7a1c3bfd5ab0bdae451512f6cf25f64ed37f5b87c",
                "sha256:acad2d8b20a1af07d4e4c9d2e9285c5ed9104354062f275f3fcd88dcef4f1326"
            ],
            "index": "pypi",
            "version": "==2021.3"
        },
        "wrapt": {
            "editable": true,
            "path": "."
        }
    },
    "develop": {}
}
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4686$ diff Pipfile.lock Pipfile.lock.initial