Any package requiring importlib-metadata breaks locking #4476

jobevers · 2020-09-28T18:29:29Z

The most recent version of importlib-metadata is 2.0.0. This is the version that gets resolved when any package is installed that required importlib-metadata without an upper bound. For example: flake8.

The virtualenv library requires importlib-metadata < 2.0.

When project.get_environment() is called the dependencies for virtualenv are compared to the working set resolved for the Pipfile dependencies and a VersonConflict error is raised.

This is a very frustrating error because doing pipenv lock -v gives no indication of which package requires importlib-metadata < 2.0. Also inspecting pipenv graph gives no indication either.

A temporary fix would be to pin importlib-metadata == "1.7.0" in the Pipfile.

The text was updated successfully, but these errors were encountered:

meshantz · 2020-09-29T16:07:55Z

I'm encountering this too. A bit more information: this seems to happen when the upper bound is only set in the dev-packages section.

In the case of this Pipfile importlib-metadata gets locked to version 2.0.0:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
virtualenv = "==20.0.31"

[packages]
flake8 = "==3.8.3"

[requires]
python_version = "3.6"

However, with this file, importlib-metadata gets properly locked to version 1.7.0:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
virtualenv = "==20.0.31"
flake8 = "==3.8.3"

[requires]
python_version = "3.6"

epicserve · 2020-09-30T15:40:41Z

Here are my debugging notes. Still not sure which package is causing the problem, but this does seem to fix it.

The error you get when you first run pipenv install with pipenv version 2020.8.13.

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 252, in install
    site_packages=state.site_packages
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1928, in do_install
    site_packages=site_packages,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 580, in ensure_project
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 512, in ensure_virtualenv
    python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 999, in do_create_virtualenv
    project._environment.add_dist("pipenv")
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 135, in add_dist
    self.extend_dists(dist)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 642, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (importlib-metadata 2.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('importlib-metadata<2,>=0.12; python_version < "3.8"'))

If you run pip install -U pipenv it seems to change the importlib-metadata version:

Installing collected packages: importlib-metadata
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 2.0.0
    Uninstalling importlib-metadata-2.0.0:
      Successfully uninstalled importlib-metadata-2.0.0
Successfully installed importlib-metadata-1.7.0

Now if you run pipenv install -d --skip-lock it will finish. It seems like a library is requiring a version >= importlib-metadata 2.0.

When I pinned the following dependencies it didn't work at first when running pipenv lock, however, if I removed the lock file (rm Pipenv.lock) then it worked when I ran pipenv lock again.

virtualenv = "==20.0.31"
importlib-metadata = "==1.7.0"

JustBob81 · 2020-10-01T14:28:26Z

The sequence of calls causing this error are fairly elaborate.

Call pipenv lock
This results in a call to main in resolver.py within a subprocess
This causes a call to clean_results only if there is an already existing lock file
This results in an attempt to instantiate the project environment to find reverse dependencies.
When instantiating the project environment the pipenv distribution is added to the environment using extend_dists
In extend_dists a search is conducted for all dependencies of pipenv in the working set
This results in a search for package versions matching the requirements of virtualenv including importlib-metadata

If this search results in finding no version of the package installed, then there is no error.

However, if there is only a version of the package that conflicts with virtualenv's requirements installed then the error discussed above is thrown.

So to see this specific error you would likely need to fall under the following conditions:

Use importlib-metadata in your project (directly or as a dependency)
Use a version of python in your project for which virtualenv requires importlib-metadata <2 (e.g. 3.7)
Not have importlib-metadata 1.7 or earlier installed elsewhere so that there is a non-conflicting version of the package in the working set.

Without fully understanding the logic of adding pipenv to the project environment, it is hard to know the right point at which to address this issue.

However, it would seem that the dependency check for pipenv should use the version of python which will be used to run pipenv. Then an error would reflect a real problem, rather than the current situation.

Alternatively, error handling around adding pipenv to the environment could handle VersionConflict errors. Either silently, which would assume that pipenv was correctly installed with its dependencies, but that might obviate the point of adding it to the environment. Or perhaps by installing required dependencies for pipenv for the project version of python.

nicofuchsg · 2020-10-01T20:52:52Z

With pip install -U pipenv was solved for me.

joshsleeper · 2020-10-02T17:59:12Z

@nicofuchsg if you're implying that simply running the latest version of pipenv solves the problem, fyi that's not the case universally.

I encountered this issue using pipenv 2020.8.13 from start to finish.

epicserve · 2020-10-03T16:25:37Z

If people go back and look at my notes it confirms that simply running pip install -U pipenv did not work for me.

cc: @nicofuchsg

DriverX · 2020-10-07T09:28:02Z

pip uninstall -y importlib-metadata and run pipenv lock again work for me.

epicserve · 2020-10-08T19:12:06Z

@DriverX, that doesn't work for me, unfortunately. The only thing that works is what I documented above.

joshsleeper · 2020-10-08T20:35:26Z

I found that my issue was actually just matter of updating to the newest virtualenv patch release 20.0.32

once that version was released and incorporated into my lockfiles everything worked like a charm on both python 3.7 and 3.8.

scuerda · 2020-10-14T18:24:24Z

The workaround of deleting the Pipfile.lock file and then re-running pipenv lock always seems to work but here is what I did for a more permanent fix.

Destroy local pipenv virtualenv pipenv --rm
Uninstall pipenv brew uninstall pipenv
Upgrade virtualenv pip install -U virtualenv
Reinstall pipenv pip install --user pipenv
Recreate pipenv virtualenv pipenv install

Repeat runs of pipenv lock work as expected and do not re-trigger the conflict error message.

* Fixes version for `importlib-metadata` to a smaller version than 2.0 to avoid an error when deploying to clusters. See pypa/pipenv#4476 for more info. Co-Authored-by: Sébastien Délèze <[email protected]>

Bobronium · 2020-10-22T23:31:57Z

@scuerda, worked like a charm, thank you!

frostming · 2020-10-27T10:05:02Z

This issue is fixed by the newest release of virtualenv

Any newcomers to this issue should upgrade the version of virtualenv

abdpp · 2020-11-03T20:39:55Z

How about if we are using pip instead of virtualenv.

saisankargochhayat · 2020-11-23T20:39:43Z

With importlib-metadata 3.0.0 this error seems to be occurring again, even after having the latest version of virtualenv

⠹ Locking..✘ Locking Failed! 
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 714, in resolve_packages
    results = clean_results(results, resolver, project, dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 590, in clean_results
    reverse_deps = project.environment.reverse_dependencies()
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/project.py", line 376, in environment
    self._environment = self.get_environment(allow_global=allow_global)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/project.py", line 366, in get_environment
    environment.extend_dists(pipenv_dist)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/tmp/tmppbnfl0ad/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 627, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (importlib-metadata 3.0.0 (/tmp/tmppbnfl0ad/.venv/lib/python3.6/site-packages), Requirement.parse('importlib-metadata<3,>=0.12; python_version < "3.8"'))

Any suggestions on what we could do to avoid this?
@joshsleeper

neverfox · 2020-12-01T19:14:00Z

I only ran into this issue when I was using a pyenv miniconda version for my pipenv's python. No issues when I use a standard 3.8 pyenv installation.

CarlosVazquezStylesage · 2020-12-04T08:33:38Z

Following @scuerda comment it works for me to fix importlib-metadata 3.0.0 with last pipenv version, hope it helps

The workaround of deleting the Pipfile.lock file and then re-running pipenv lock always seems to work but here is what I did for a more permanent fix.

Destroy local pipenv virtualenv pipenv --rm

Uninstall pipenv brew uninstall pipenv

Upgrade virtualenv pip install -U virtualenv

Reinstall pipenv pip install --user pipenv

Recreate pipenv virtualenv pipenv install

Repeat runs of pipenv lock work as expected and do not re-trigger the conflict error message.

a-froghyar · 2020-12-13T15:05:49Z

I'm still having this issue trying to install tensorflow 1.15 with pip on OS X into a conda environment. I install a previous version of importlib-metadata however one of the dependencies in the tensorflow package is forcing the newer version of importlib-metadata:

Installing collected packages: zipp, six, numpy, importlib-metadata, cached-property, wheel, werkzeug, setuptools, protobuf, markdown, h5py, grpcio, absl-py, wrapt, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, keras-applications, google-pasta, gast, astor, tensorflow

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. virtualenv 20.0.31 requires importlib-metadata<2,>=0.12; python_version < "3.8", but you have importlib-metadata 3.1.1 which is incompatible.

EDIT: I've managed to install tensorflow 1.15 using conda.

januszm · 2021-01-08T18:14:13Z

I wonder if there are plans for the pipenv to use the new built-in pip dependency resolver.

NeroVanbiervliet · 2021-04-27T07:01:04Z

I was using multiple versions of python on my system (3.8 as main, but also 3.7).

So instead of upgrading pipenv as suggested above like this

pip3 -U pipenv

I upgraded pipenv for the version of python that was used by my project:

python3.7 -m pip install -U pipenv

- We added only the necessary packages in the Pipfile (there were more packages added there by mistake). We had to pin the flake8 package because of the following error when installing the packages: ERROR: Cannot install -r /home/user/dev/grades-report/dev-requirements.txt (line 6), -r /home/user/dev/grades-report/requirements.txt (line 2) and importlib-metadata==4.12.0 because these package versions have conflicting dependencies. This seems to be related to pypa/pipenv#4476. - Added an option in the Makefile to create the latest requirements files with the latest available package versions.

Update README.md: Installation Guidelines 1. Added change to correct directory for installation 2. While running installation guideline for raw37 and py37, encountered this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. flake8 4.0.1 requires importlib-metadata<4.3; python_version < "3.8", but you have importlib-metadata 6.0.0 which is incompatible. flake8 4.0.1 requires mccabe<0.7.0,>=0.6.0, but you have mccabe 0.7.0 which is incompatible. This is discussed in a github issue: pypa/pipenv#4476 Added Fix : use command: pip install -U pipenv

jobevers mentioned this issue Sep 28, 2020

lock works first time, fails when tried again #4469

Closed

sebdeleze mentioned this issue Oct 20, 2020

dependencies: fix version for importlib-metadata rero/sonar#360

Merged

frostming closed this as completed Oct 27, 2020

saisankargochhayat mentioned this issue Nov 23, 2020

Updated virtualenv thoth-station/kebechet#610

Closed

frostming reopened this Nov 24, 2020

frostming added the Category: Dependency Resolution Issue relates to dependency resolution. label Nov 24, 2020

frostming mentioned this issue Jan 20, 2021

Fix #4549: Ignore incompatibilities when resolve dist #4593

Merged

2 tasks

frostming closed this as completed in #4593 Jan 20, 2021

frostming mentioned this issue Jan 22, 2021

pkg_resources.VersionConflict: (importlib-metadata 2.1.1 (/home/salim/.local/lib/python3.7/site-packages), Requirement.parse('importlib-metadata<2,>=0.12; python_version < "3.8"')) #4597

Closed

andrewdotn mentioned this issue Jan 29, 2021

Page for phrase UAlbertaALTLab/recording-validation-interface#106

Merged

Turbo87 mentioned this issue Jan 31, 2021

Docker: Upgrade virtualenv before installing pipenv skylines-project/skylines#2377

Merged

oliver-leung mentioned this issue Feb 2, 2021

Migrate from Travis CI and Appveyor to Github Actions AguaClara/aguaclara#292

Merged

KAUTH mentioned this issue Jul 23, 2022

Correct the Pipfile content and update packages KAUTH/grades-report#10

Merged

KAUTH mentioned this issue Aug 25, 2022

Add pre-commit tool OwnCA/ownca#77

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Any package requiring importlib-metadata breaks locking #4476

Any package requiring importlib-metadata breaks locking #4476

jobevers commented Sep 28, 2020

meshantz commented Sep 29, 2020 •

edited

Loading

epicserve commented Sep 30, 2020 •

edited

Loading

JustBob81 commented Oct 1, 2020

nicofuchsg commented Oct 1, 2020

joshsleeper commented Oct 2, 2020

epicserve commented Oct 3, 2020

DriverX commented Oct 7, 2020

epicserve commented Oct 8, 2020

joshsleeper commented Oct 8, 2020

scuerda commented Oct 14, 2020

Bobronium commented Oct 22, 2020

frostming commented Oct 27, 2020

abdpp commented Nov 3, 2020

saisankargochhayat commented Nov 23, 2020 •

edited

Loading

neverfox commented Dec 1, 2020

CarlosVazquezStylesage commented Dec 4, 2020

a-froghyar commented Dec 13, 2020 •

edited

Loading

januszm commented Jan 8, 2021

NeroVanbiervliet commented Apr 27, 2021

Any package requiring importlib-metadata breaks locking #4476

Any package requiring importlib-metadata breaks locking #4476

Comments

jobevers commented Sep 28, 2020

meshantz commented Sep 29, 2020 • edited Loading

epicserve commented Sep 30, 2020 • edited Loading

JustBob81 commented Oct 1, 2020

nicofuchsg commented Oct 1, 2020

joshsleeper commented Oct 2, 2020

epicserve commented Oct 3, 2020

DriverX commented Oct 7, 2020

epicserve commented Oct 8, 2020

joshsleeper commented Oct 8, 2020

scuerda commented Oct 14, 2020

Bobronium commented Oct 22, 2020

frostming commented Oct 27, 2020

abdpp commented Nov 3, 2020

saisankargochhayat commented Nov 23, 2020 • edited Loading

neverfox commented Dec 1, 2020

CarlosVazquezStylesage commented Dec 4, 2020

a-froghyar commented Dec 13, 2020 • edited Loading

januszm commented Jan 8, 2021

NeroVanbiervliet commented Apr 27, 2021

meshantz commented Sep 29, 2020 •

edited

Loading

epicserve commented Sep 30, 2020 •

edited

Loading

saisankargochhayat commented Nov 23, 2020 •

edited

Loading

a-froghyar commented Dec 13, 2020 •

edited

Loading