Uninstall of python-dotenv does not (really) work

[soraphis]

Issue description

I also tried reaching out on stackoverflow, but at this point. it feels like a bug

version: pipenv, version 2023.3.20

I try to uninstall python-dotenv. Should be easy, right? It's not in my Pipfile, and in my pipenv-graph it's not listed as a transitive dependency.

Following this series of commands:

$ echo "graph"; pipenv graph | grep python-dotenv; echo "Pipfile.lock"; cat Pipfile.lock | grep python-dotenv; echo "Pipfile"; cat Pipfile | grep python-dotenv   
graph
Loading .env environment variables...
python-dotenv==1.0.0
Pipfile.lock
        "python-dotenv": {
Pipfile
$ pipenv uninstall python-dotenv
Loading .env environment variables...
Removing python-dotenv from Pipfile.lock...
Uninstalling python-dotenv...
Found existing installation: python-dotenv 1.0.0
Uninstalling python-dotenv-1.0.0:
  Successfully uninstalled python-dotenv-1.0.0

Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...                                                                                                                                                                                                                               
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (9362bc95e99217f2f5cac841a8b99b7b491e5c52886bf094ff10df3874969e05)!                                                                                                                                                                             
$ echo "graph"; pipenv graph | grep python-dotenv; echo "Pipfile.lock"; cat Pipfile.lock | grep python-dotenv; echo "Pipfile"; cat Pipfile | grep python-dotenv   
graph
Loading .env environment variables...
Pipfile.lock
        "python-dotenv": {
Pipfile

notice, how it's NOT deleted from the Pipfile.lock?, well, if i now run "pipenv install", I went full circle and have it back in my pipenv graph.

I thought removing it manually from the pipfile lock would work, but after running "pipenv lock" (or any other command that recreates the lockfile) it's back in there again, even if it was not present in the Pipfile nor pipenv graph.

Expected result

the package being actually removed, and not reappearing again

---

pipenv --support is way to large to paste here
pipenv__support.txt

[matteius]

@soraphis Ok I spent a bit debugging this, and it seems it is because you have django-dotenv = "*" in your Pipfile which certainly requires python-dotenv -- I am not sure why its showing up as a top level item in the graph though.

[matteius]

Sorry, I was slightly off base in my last response, the real reason its included is uvicorn: https://github.com/encode/uvicorn/blob/master/pyproject.toml#L41

[soraphis]

Thanks for taking the time and looking into it. What I don't really get is: why isn't showing up in the graph? Other dependencies are also listed multiple times if required by multiple packages...

[derula]

For the record, I tested this the other day out of curiosity. PDM also installs python-dotenv when importing from this Pipfile, but doesn't list it in pdm list --graph at all, which I thought was very strange.

[matteius]

The new version of pipdeptree appears to fix this, I have a PR out to update pipenv.

matteius@matteius-VirtualBox:~/pipenv-triage/issue-5652$ pipenv graph
autopep8==2.0.2
├── pycodestyle [required: >=2.10.0, installed: 2.10.0]
└── tomli [required: Any, installed: 2.0.1]
better-exceptions==0.3.3
channels==4.0.0
├── asgiref [required: >=3.5.0,<4, installed: 3.7.2]
│   └── typing-extensions [required: >=4, installed: 4.6.3]
└── Django [required: >=3.2, installed: 3.2]
    ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
    │   └── typing-extensions [required: >=4, installed: 4.6.3]
    ├── pytz [required: Any, installed: 2023.3]
    └── sqlparse [required: >=0.2.2, installed: 0.4.4]
daemonocle==1.2.3
├── click [required: Any, installed: 8.1.3]
└── psutil [required: Any, installed: 5.9.5]
django-allauth==0.54.0
├── Django [required: >=2.0, installed: 3.2]
│   ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
│   │   └── typing-extensions [required: >=4, installed: 4.6.3]
│   ├── pytz [required: Any, installed: 2023.3]
│   └── sqlparse [required: >=0.2.2, installed: 0.4.4]
├── pyjwt [required: >=1.7, installed: 2.7.0]
├── python3-openid [required: >=3.0.8, installed: 3.2.0]
│   └── defusedxml [required: Any, installed: 0.7.1]
├── requests [required: Any, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
│   ├── idna [required: >=2.5,<4, installed: 3.4]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.0.2]
└── requests-oauthlib [required: >=0.3.0, installed: 1.3.1]
    ├── oauthlib [required: >=3.0.0, installed: 3.2.2]
    └── requests [required: >=2.0.0, installed: 2.31.0]
        ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
        ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
        ├── idna [required: >=2.5,<4, installed: 3.4]
        └── urllib3 [required: >=1.21.1,<3, installed: 2.0.2]
django-anymail==10.0
├── cryptography [required: Any, installed: 41.0.1]
│   └── cffi [required: >=1.12, installed: 1.15.1]
│       └── pycparser [required: Any, installed: 2.21]
├── django [required: >=2.0, installed: 3.2]
│   ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
│   │   └── typing-extensions [required: >=4, installed: 4.6.3]
│   ├── pytz [required: Any, installed: 2023.3]
│   └── sqlparse [required: >=0.2.2, installed: 0.4.4]
├── requests [required: >=2.4.3, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
│   ├── idna [required: >=2.5,<4, installed: 3.4]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.0.2]
└── urllib3 [required: >=1.25.0, installed: 2.0.2]
django-cron==0.6.0
└── Django [required: >=3.2, installed: 3.2]
    ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
    │   └── typing-extensions [required: >=4, installed: 4.6.3]
    ├── pytz [required: Any, installed: 2023.3]
    └── sqlparse [required: >=0.2.2, installed: 0.4.4]
django-dotenv==1.4.2
django-filter==23.2
└── Django [required: >=3.2, installed: 3.2]
    ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
    │   └── typing-extensions [required: >=4, installed: 4.6.3]
    ├── pytz [required: Any, installed: 2023.3]
    └── sqlparse [required: >=0.2.2, installed: 0.4.4]
django-getenv==1.3.2
django-guardian==2.4.0
└── Django [required: >=2.2, installed: 3.2]
    ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
    │   └── typing-extensions [required: >=4, installed: 4.6.3]
    ├── pytz [required: Any, installed: 2023.3]
    └── sqlparse [required: >=0.2.2, installed: 0.4.4]
django-hashids==0.7.0
└── hashids [required: >=1.0.2, installed: 1.3.1]
django-push-notifications==3.0.0
└── Django [required: >=2.2, installed: 3.2]
    ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
    │   └── typing-extensions [required: >=4, installed: 4.6.3]
    ├── pytz [required: Any, installed: 2023.3]
    └── sqlparse [required: >=0.2.2, installed: 0.4.4]
django-stubs==4.2.1
├── django [required: Any, installed: 3.2]
│   ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
│   │   └── typing-extensions [required: >=4, installed: 4.6.3]
│   ├── pytz [required: Any, installed: 2023.3]
│   └── sqlparse [required: >=0.2.2, installed: 0.4.4]
├── django-stubs-ext [required: >=4.2.1, installed: 4.2.1]
│   ├── django [required: Any, installed: 3.2]
│   │   ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
│   │   │   └── typing-extensions [required: >=4, installed: 4.6.3]
│   │   ├── pytz [required: Any, installed: 2023.3]
│   │   └── sqlparse [required: >=0.2.2, installed: 0.4.4]
│   └── typing-extensions [required: Any, installed: 4.6.3]
├── mypy [required: >=1.0.0, installed: 1.3.0]
│   ├── mypy-extensions [required: >=1.0.0, installed: 1.0.0]
│   ├── tomli [required: >=1.1.0, installed: 2.0.1]
│   └── typing-extensions [required: >=3.10, installed: 4.6.3]
├── tomli [required: Any, installed: 2.0.1]
├── types-pytz [required: Any, installed: 2023.3.0.0]
├── types-PyYAML [required: Any, installed: 6.0.12.10]
└── typing-extensions [required: Any, installed: 4.6.3]
django-widget-tweaks==1.4.12
djangorestframework==3.14.0
├── django [required: >=3.0, installed: 3.2]
│   ├── asgiref [required: >=3.3.2,<4, installed: 3.7.2]
│   │   └── typing-extensions [required: >=4, installed: 4.6.3]
│   ├── pytz [required: Any, installed: 2023.3]
│   └── sqlparse [required: >=0.2.2, installed: 0.4.4]
└── pytz [required: Any, installed: 2023.3]
httptools==0.5.0
jsonschema==4.17.3
├── attrs [required: >=17.4.0, installed: 23.1.0]
├── importlib-resources [required: >=1.4.0, installed: 5.12.0]
│   └── zipp [required: >=3.1.0, installed: 3.15.0]
├── pkgutil-resolve-name [required: >=1.3.10, installed: 1.3.10]
└── pyrsistent [required: >=0.14.0,!=0.17.2,!=0.17.1,!=0.17.0, installed: 0.19.3]
Markdown==3.4.3
└── importlib-metadata [required: >=4.4, installed: 6.6.0]
    └── zipp [required: >=0.5, installed: 3.15.0]
pylint-django==2.5.3
├── pylint [required: >=2.0,<3, installed: 2.17.4]
│   ├── astroid [required: >=2.15.4,<=2.17.0-dev0, installed: 2.15.5]
│   │   ├── lazy-object-proxy [required: >=1.4.0, installed: 1.9.0]
│   │   ├── typing-extensions [required: >=4.0.0, installed: 4.6.3]
│   │   └── wrapt [required: >=1.11,<2, installed: 1.15.0]
│   ├── dill [required: >=0.2, installed: 0.3.6]
│   ├── isort [required: >=4.2.5,<6, installed: 5.12.0]
│   ├── mccabe [required: >=0.6,<0.8, installed: 0.7.0]
│   ├── platformdirs [required: >=2.2.0, installed: 3.5.1]
│   ├── tomli [required: >=1.1.0, installed: 2.0.1]
│   ├── tomlkit [required: >=0.10.1, installed: 0.11.8]
│   └── typing-extensions [required: >=3.10.0, installed: 4.6.3]
└── pylint-plugin-utils [required: >=0.7, installed: 0.8.2]
    └── pylint [required: >=1.7, installed: 2.17.4]
        ├── astroid [required: >=2.15.4,<=2.17.0-dev0, installed: 2.15.5]
        │   ├── lazy-object-proxy [required: >=1.4.0, installed: 1.9.0]
        │   ├── typing-extensions [required: >=4.0.0, installed: 4.6.3]
        │   └── wrapt [required: >=1.11,<2, installed: 1.15.0]
        ├── dill [required: >=0.2, installed: 0.3.6]
        ├── isort [required: >=4.2.5,<6, installed: 5.12.0]
        ├── mccabe [required: >=0.6,<0.8, installed: 0.7.0]
        ├── platformdirs [required: >=2.2.0, installed: 3.5.1]
        ├── tomli [required: >=1.1.0, installed: 2.0.1]
        ├── tomlkit [required: >=0.10.1, installed: 0.11.8]
        └── typing-extensions [required: >=3.10.0, installed: 4.6.3]
PyMonad==2.4.0
python-dotenv==1.0.0
pywebpush==1.14.0
├── cryptography [required: >=2.6.1, installed: 41.0.1]
│   └── cffi [required: >=1.12, installed: 1.15.1]
│       └── pycparser [required: Any, installed: 2.21]
├── http-ece [required: >=1.1.0, installed: 1.1.0]
│   └── cryptography [required: >=2.5, installed: 41.0.1]
│       └── cffi [required: >=1.12, installed: 1.15.1]
│           └── pycparser [required: Any, installed: 2.21]
├── py-vapid [required: >=1.7.0, installed: 1.9.0]
│   └── cryptography [required: >=2.5, installed: 41.0.1]
│       └── cffi [required: >=1.12, installed: 1.15.1]
│           └── pycparser [required: Any, installed: 2.21]
├── requests [required: >=2.21.0, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
│   ├── idna [required: >=2.5,<4, installed: 3.4]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.0.2]
└── six [required: >=1.15.0, installed: 1.16.0]
PyYAML==6.0
uvicorn==0.22.0
├── click [required: >=7.0, installed: 8.1.3]
└── h11 [required: >=0.8, installed: 0.14.0]
uvloop==0.17.0
watchfiles==0.19.0
└── anyio [required: >=3.0.0, installed: 3.7.0]
    ├── exceptiongroup [required: Any, installed: 1.1.1]
    ├── idna [required: >=2.8, installed: 3.4]
    └── sniffio [required: >=1.1, installed: 1.3.0]
websockets==11.0.3
whitenoise==6.4.0

[matteius]

Well it kind of fixes it -- it shows up in the graph, just not under uvicorn, which seems odd.

[derula]

Whether it appears in the graph is kind of secondary, I think. The real problem here is that:

• python_dotenv is an optional dependency of uvicorn
• python_dotenv and dotenv are incompatible

Thus:

• Nothing strictly requires python_dotenv (at least uvicorn doesn't, I haven't checked all other packages)
• python_dotenv is installed anyway (for what reason?)
• dotenv package breaks / may break as a result

IIRC, pipenv graph always showed python_dotenv as a top-level dependency, not under uvicorn.

Pure speculation: maybe the resolver includes all optional dependencies, thus python_dotenv is installed, but the dependency tree does not consider optional dependencies, thus it is not listed as a dependency of any other package.

[matteius]

Actually that part is easy to explain @derula --

uvicorn = {extras = ["standard"], version = "*"}

extras = ["standard"] does tell it to install all of: https://github.com/encode/uvicorn/blob/master/pyproject.toml#L38-L46

[derula]

Ah, okay, I must have missed that. Then I guess there are 2 separate problems:

• @soraphis shouldn't be using the standard extra because that will cause python-dotenv to be installed, conflicting with dotenv
• python-dotenv (maybe all optional deps?) not listed as sub-dependency of uvicorn

[matteius]

python-dotenv (maybe all optional deps?) not listed as sub-dependency of uvicorn

@derula that should be reported upstream to pipdeptree

[derula]

@derula that should be reported upstream to pipdeptree

Upstream ticket has existed for over 4 years: tox-dev/pipdeptree/issues/107
PR has since been created, diverged, and closed: tox-dev/pipdeptree/pull/138
Seems like it won't be fixing itself :)