Performance of the new resolver when installing/upgrading many packages at once

[bersbersbers]

What did you want to do?
I liked to use

pip list --outdated --format freeze | sed 's/==.*//' | xargs --no-run-if-empty pip install --upgrade --upgrade-strategy eager

to update all packages. However, this updates packages beyond the requirements of installed packages, which pip ignores, leading to conflicts.

Same thing even for

pip list --format freeze | sed 's/==.*//' | xargs --no-run-if-empty pip install --upgrade --upgrade-strategy eager

Now, the new resolver looks promising in the second case (it still ignores installed packages, so --outdated is not an option, but that's not too much of an issue):

pip list --format freeze | sed 's/==.*//' | xargs --no-run-if-empty pip install --upgrade --upgrade-strategy eager --use-feature=2020-resolver

It upgrades only what's possible to upgrade without causing conflicts, and even downgrades what's required to resolve conflicts. It's basically a one-stop-shop to fix a set of installed packages. Nicely done!

However, it's slow:

• pip list --outdated takes ~15s
• pip list | sed | xargs pip install takes ~9s with the old resolver (in its 2nd run when all package installation have finished), but it takes ~55s with the new resolver (again, in its 2nd run without package installation)

I'm surprised that pip list | sed | xargs pip install is faster than pip list --outdated because I expect the latter to do much more than the former, but that's not what I am worried about. But note how much slower pip list | sed | xargs pip install is with the new resolver. I first suspected additional (repeated) HTTP requests for the same packages as an issue, but

pip list --format freeze | sed 's/==.*//' | xargs --no-run-if-empty pip install --upgrade --upgrade-strategy eager --use-feature=2020-resolver -v | grep GET | sort

shows the same (125) requests as

pip list --outdated -v | grep GET | sort

Additional information

absl-py==0.11.0
appdirs==1.4.4
astroid==2.4.2
astunparse==1.6.3
bandit==1.6.2
black==20.8b1
blessed==1.17.11
cachetools==4.1.1
certifi==2020.11.8
chardet==3.0.4
-e [email protected]:user/myproj.git@hash#egg=myproj
click==7.1.2
colour-science==0.3.15
cycler==0.10.0
decorator==4.4.2
descartes==1.1.0
docopt==0.6.2
ExifRead==2.3.2
flake8==3.8.4
flatbuffers==1.12
flyingcircus==0.1.3.2
flyingcircus-numeric==0.1.1.2
gast==0.3.3
gitdb==4.0.5
GitPython==3.1.11
google-auth==1.23.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
humanize==3.1.0
idna==2.10
imageio==2.9.0
isort==5.6.4
joblib==0.17.0
Keras==2.4.3
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
lazy-object-proxy==1.4.3
lxml==4.6.1
Markdown==3.3.3
matplotlib==3.3.3
mccabe==0.6.1
mizani==0.7.2
mypy==0.790
mypy-extensions==0.4.3
natsort==7.0.1
networkx==2.5
nose==1.3.7
numpy==1.19.4
oauthlib==3.1.0
opencv-contrib-python==4.4.0.46
opencv-python==4.4.0.46
opt-einsum==3.3.0
ordered-set==4.0.2
palettable==3.3.0
pandas==1.1.4
pathspec==0.8.1
patsy==0.5.1
pbr==5.5.1
Pillow==8.0.1
pipdeptree==1.0.0
pipreqs==0.4.10
plotnine==0.7.1
protobuf==3.13.0
psutil==5.7.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.6.0
pydocstyle==5.1.1
pyflakes==2.2.0
pylama==7.7.1
pylint==2.6.0
pynrrd==0.4.2
pyparsing==2.4.7
python-dateutil==2.8.1
python-pptx==0.6.18
pytz==2020.4
PyWavelets==1.1.1
PyYAML==5.3.1
raster-geometry==0.1.4.1
regex==2020.11.11
requests==2.25.0
requests-oauthlib==1.3.0
rope==0.18.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.2
scikit-plot==0.3.7
scipy==1.5.4
seaborn==0.11.0
setuptools-scm==4.1.2
SharedArray==3.2.1
silence-tensorflow==1.1.1
six==1.15.0
smmap==3.0.4
snowballstemmer==2.0.0
statsmodels==0.12.1
stevedore==3.2.2
temppathlib==1.0.4
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.4.0rc1
tensorflow-addons==0.11.2
tensorflow-estimator==2.4.0rc0
termcolor==1.1.0
tf-explain==0.2.1
tf-keras-vis==0.5.3
threadpoolctl==2.1.0
tifffile==2020.10.1
toml==0.10.2
toolz==0.11.1
typed-ast==1.4.1
typeguard==2.10.0
typing-extensions==3.7.4.3
urllib3==1.26.1
wcwidth==0.2.5
Werkzeug==1.0.1
wrapt==1.12.1
xlrd==1.2.0
XlsxWriter==1.3.7
yarg==0.1.9

bandit==1.6.2
  - GitPython [required: >=1.0.1, installed: 3.1.11]
    - gitdb [required: >=4.0.1,<5, installed: 4.0.5]
      - smmap [required: >=3.0.1,<4, installed: 3.0.4]
  - PyYAML [required: >=3.13, installed: 5.3.1]
  - six [required: >=1.10.0, installed: 1.15.0]
  - stevedore [required: >=1.20.0, installed: 3.2.2]
    - pbr [required: >=2.0.0,!=2.1.0, installed: 5.5.1]
black==20.8b1
  - appdirs [required: Any, installed: 1.4.4]
  - click [required: >=7.1.2, installed: 7.1.2]
  - mypy-extensions [required: >=0.4.3, installed: 0.4.3]
  - pathspec [required: >=0.6,<1, installed: 0.8.1]
  - regex [required: >=2020.1.8, installed: 2020.11.11]
  - toml [required: >=0.10.1, installed: 0.10.2]
  - typed-ast [required: >=1.4.0, installed: 1.4.1]
  - typing-extensions [required: >=3.7.4, installed: 3.7.4.3]
myproj==0.1
colour-science==0.3.15
  - imageio [required: Any, installed: 2.9.0]
    - numpy [required: Any, installed: 1.19.4]
    - pillow [required: Any, installed: 8.0.1]
  - scipy [required: Any, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
  - six [required: Any, installed: 1.15.0]
ExifRead==2.3.2
flake8==3.8.4
  - mccabe [required: >=0.6.0,<0.7.0, installed: 0.6.1]
  - pycodestyle [required: >=2.6.0a1,<2.7.0, installed: 2.6.0]
  - pyflakes [required: >=2.2.0,<2.3.0, installed: 2.2.0]
humanize==3.1.0
  - setuptools [required: Any, installed: 50.3.2]
Keras==2.4.3
  - h5py [required: Any, installed: 2.10.0]
    - numpy [required: >=1.7, installed: 1.19.4]
    - six [required: Any, installed: 1.15.0]
  - numpy [required: >=1.9.1, installed: 1.19.4]
  - pyyaml [required: Any, installed: 5.3.1]
  - scipy [required: >=0.14, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
mypy==0.790
  - mypy-extensions [required: >=0.4.3,<0.5.0, installed: 0.4.3]
  - typed-ast [required: >=1.4.0,<1.5.0, installed: 1.4.1]
  - typing-extensions [required: >=3.7.4, installed: 3.7.4.3]
natsort==7.0.1
nose==1.3.7
opencv-contrib-python==4.4.0.46
  - numpy [required: >=1.17.3, installed: 1.19.4]
ordered-set==4.0.2
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.4]
pipreqs==0.4.10
  - docopt [required: Any, installed: 0.6.2]
  - yarg [required: Any, installed: 0.1.9]
    - requests [required: Any, installed: 2.25.0]
      - certifi [required: >=2017.4.17, installed: 2020.11.8]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.1]
plotnine==0.7.1
  - descartes [required: >=1.1.0, installed: 1.1.0]
    - matplotlib [required: Any, installed: 3.3.3]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.19.4]
      - pillow [required: >=6.2.0, installed: 8.0.1]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
  - matplotlib [required: >=3.1.1, installed: 3.3.3]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.19.4]
    - pillow [required: >=6.2.0, installed: 8.0.1]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - mizani [required: >=0.7.1, installed: 0.7.2]
    - matplotlib [required: >=3.1.1, installed: 3.3.3]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.19.4]
      - pillow [required: >=6.2.0, installed: 8.0.1]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
    - numpy [required: Any, installed: 1.19.4]
    - palettable [required: Any, installed: 3.3.0]
    - pandas [required: >=1.1.0, installed: 1.1.4]
      - numpy [required: >=1.15.4, installed: 1.19.4]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.2, installed: 2020.4]
  - numpy [required: >=1.16.0, installed: 1.19.4]
  - pandas [required: >=1.1.0, installed: 1.1.4]
    - numpy [required: >=1.15.4, installed: 1.19.4]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.4]
  - patsy [required: >=0.5.1, installed: 0.5.1]
    - numpy [required: >=1.4, installed: 1.19.4]
    - six [required: Any, installed: 1.15.0]
  - scipy [required: >=1.2.0, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
  - statsmodels [required: >=0.11.1, installed: 0.12.1]
    - numpy [required: >=1.15, installed: 1.19.4]
    - pandas [required: >=0.21, installed: 1.1.4]
      - numpy [required: >=1.15.4, installed: 1.19.4]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.2, installed: 2020.4]
    - patsy [required: >=0.5, installed: 0.5.1]
      - numpy [required: >=1.4, installed: 1.19.4]
      - six [required: Any, installed: 1.15.0]
    - scipy [required: >=1.1, installed: 1.5.4]
      - numpy [required: >=1.14.5, installed: 1.19.4]
psutil==5.7.3
pylama==7.7.1
  - mccabe [required: >=0.5.2, installed: 0.6.1]
  - pycodestyle [required: >=2.3.1, installed: 2.6.0]
  - pydocstyle [required: >=2.0.0, installed: 5.1.1]
    - snowballstemmer [required: Any, installed: 2.0.0]
  - pyflakes [required: >=1.5.0, installed: 2.2.0]
pylint==2.6.0
  - astroid [required: >=2.4.0,<=2.5, installed: 2.4.2]
    - lazy-object-proxy [required: ==1.4.*, installed: 1.4.3]
    - six [required: ~=1.12, installed: 1.15.0]
    - wrapt [required: ~=1.11, installed: 1.12.1]
  - isort [required: >=4.2.5,<6, installed: 5.6.4]
  - mccabe [required: >=0.6,<0.7, installed: 0.6.1]
  - toml [required: >=0.7.1, installed: 0.10.2]
pynrrd==0.4.2
  - numpy [required: >=1.11.1, installed: 1.19.4]
python-pptx==0.6.18
  - lxml [required: >=3.1.0, installed: 4.6.1]
  - Pillow [required: >=3.3.2, installed: 8.0.1]
  - XlsxWriter [required: >=0.5.7, installed: 1.3.7]
raster-geometry==0.1.4.1
  - appdirs [required: Any, installed: 1.4.4]
  - flyingcircus [required: Any, installed: 0.1.3.2]
    - appdirs [required: Any, installed: 1.4.4]
    - blessed [required: Any, installed: 1.17.11]
      - six [required: >=1.9.0, installed: 1.15.0]
      - wcwidth [required: >=0.1.4, installed: 0.2.5]
    - setuptools [required: Any, installed: 50.3.2]
    - setuptools-scm [required: Any, installed: 4.1.2]
      - setuptools [required: Any, installed: 50.3.2]
  - flyingcircus-numeric [required: Any, installed: 0.1.1.2]
    - appdirs [required: Any, installed: 1.4.4]
    - blessed [required: Any, installed: 1.17.11]
      - six [required: >=1.9.0, installed: 1.15.0]
      - wcwidth [required: >=0.1.4, installed: 0.2.5]
    - setuptools [required: Any, installed: 50.3.2]
    - setuptools-scm [required: Any, installed: 4.1.2]
      - setuptools [required: Any, installed: 50.3.2]
  - numpy [required: Any, installed: 1.19.4]
  - scipy [required: Any, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
  - setuptools [required: Any, installed: 50.3.2]
rope==0.18.0
scikit-image==0.17.2
  - imageio [required: >=2.3.0, installed: 2.9.0]
    - numpy [required: Any, installed: 1.19.4]
    - pillow [required: Any, installed: 8.0.1]
  - matplotlib [required: >=2.0.0,!=3.0.0, installed: 3.3.3]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.19.4]
    - pillow [required: >=6.2.0, installed: 8.0.1]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - networkx [required: >=2.0, installed: 2.5]
    - decorator [required: >=4.3.0, installed: 4.4.2]
  - numpy [required: >=1.15.1, installed: 1.19.4]
  - pillow [required: >=4.3.0,!=7.1.1,!=7.1.0, installed: 8.0.1]
  - PyWavelets [required: >=1.1.1, installed: 1.1.1]
    - numpy [required: >=1.13.3, installed: 1.19.4]
  - scipy [required: >=1.0.1, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
  - tifffile [required: >=2019.7.26, installed: 2020.10.1]
    - numpy [required: >=1.15.1, installed: 1.19.4]
scikit-plot==0.3.7
  - joblib [required: >=0.10, installed: 0.17.0]
  - matplotlib [required: >=1.4.0, installed: 3.3.3]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.19.4]
    - pillow [required: >=6.2.0, installed: 8.0.1]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - scikit-learn [required: >=0.18, installed: 0.23.2]
    - joblib [required: >=0.11, installed: 0.17.0]
    - numpy [required: >=1.13.3, installed: 1.19.4]
    - scipy [required: >=0.19.1, installed: 1.5.4]
      - numpy [required: >=1.14.5, installed: 1.19.4]
    - threadpoolctl [required: >=2.0.0, installed: 2.1.0]
  - scipy [required: >=0.9, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
seaborn==0.11.0
  - matplotlib [required: >=2.2, installed: 3.3.3]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.19.4]
    - pillow [required: >=6.2.0, installed: 8.0.1]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - numpy [required: >=1.15, installed: 1.19.4]
  - pandas [required: >=0.23, installed: 1.1.4]
    - numpy [required: >=1.15.4, installed: 1.19.4]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.4]
  - scipy [required: >=1.0, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
SharedArray==3.2.1
  - numpy [required: Any, installed: 1.19.4]
silence-tensorflow==1.1.1
temppathlib==1.0.4
tensorflow==2.4.0rc1
  - absl-py [required: ~=0.10, installed: 0.11.0]
    - six [required: Any, installed: 1.15.0]
  - astunparse [required: ~=1.6.3, installed: 1.6.3]
    - six [required: >=1.6.1,<2.0, installed: 1.15.0]
    - wheel [required: >=0.23.0,<1.0, installed: 0.35.1]
  - flatbuffers [required: ~=1.12.0, installed: 1.12]
  - gast [required: ==0.3.3, installed: 0.3.3]
  - google-pasta [required: ~=0.2, installed: 0.2.0]
    - six [required: Any, installed: 1.15.0]
  - grpcio [required: ~=1.32.0, installed: 1.32.0]
    - six [required: >=1.5.2, installed: 1.15.0]
  - h5py [required: ~=2.10.0, installed: 2.10.0]
    - numpy [required: >=1.7, installed: 1.19.4]
    - six [required: Any, installed: 1.15.0]
  - keras-preprocessing [required: ~=1.1.2, installed: 1.1.2]
    - numpy [required: >=1.9.1, installed: 1.19.4]
    - six [required: >=1.9.0, installed: 1.15.0]
  - numpy [required: ~=1.19.2, installed: 1.19.4]
  - opt-einsum [required: ~=3.3.0, installed: 3.3.0]
    - numpy [required: >=1.7, installed: 1.19.4]
  - protobuf [required: ~=3.13.0, installed: 3.13.0]
    - setuptools [required: Any, installed: 50.3.2]
    - six [required: >=1.9, installed: 1.15.0]
  - six [required: ~=1.15.0, installed: 1.15.0]
  - tensorboard [required: ~=2.3, installed: 2.4.0]
    - absl-py [required: >=0.4, installed: 0.11.0]
      - six [required: Any, installed: 1.15.0]
    - google-auth [required: >=1.6.3,<2, installed: 1.23.0]
      - cachetools [required: >=2.0.0,<5.0, installed: 4.1.1]
      - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
        - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
      - rsa [required: >=3.1.4,<5, installed: 4.6]
        - pyasn1 [required: >=0.1.3, installed: 0.4.8]
      - setuptools [required: >=40.3.0, installed: 50.3.2]
      - six [required: >=1.9.0, installed: 1.15.0]
    - google-auth-oauthlib [required: >=0.4.1,<0.5, installed: 0.4.2]
      - google-auth [required: Any, installed: 1.23.0]
        - cachetools [required: >=2.0.0,<5.0, installed: 4.1.1]
        - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
          - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
        - rsa [required: >=3.1.4,<5, installed: 4.6]
          - pyasn1 [required: >=0.1.3, installed: 0.4.8]
        - setuptools [required: >=40.3.0, installed: 50.3.2]
        - six [required: >=1.9.0, installed: 1.15.0]
      - requests-oauthlib [required: >=0.7.0, installed: 1.3.0]
        - oauthlib [required: >=3.0.0, installed: 3.1.0]
        - requests [required: >=2.0.0, installed: 2.25.0]
          - certifi [required: >=2017.4.17, installed: 2020.11.8]
          - chardet [required: >=3.0.2,<4, installed: 3.0.4]
          - idna [required: >=2.5,<3, installed: 2.10]
          - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.1]
    - grpcio [required: >=1.24.3, installed: 1.32.0]
      - six [required: >=1.5.2, installed: 1.15.0]
    - markdown [required: >=2.6.8, installed: 3.3.3]
    - numpy [required: >=1.12.0, installed: 1.19.4]
    - protobuf [required: >=3.6.0, installed: 3.13.0]
      - setuptools [required: Any, installed: 50.3.2]
      - six [required: >=1.9, installed: 1.15.0]
    - requests [required: >=2.21.0,<3, installed: 2.25.0]
      - certifi [required: >=2017.4.17, installed: 2020.11.8]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.1]
    - setuptools [required: >=41.0.0, installed: 50.3.2]
    - six [required: >=1.10.0, installed: 1.15.0]
    - tensorboard-plugin-wit [required: >=1.6.0, installed: 1.7.0]
    - werkzeug [required: >=0.11.15, installed: 1.0.1]
    - wheel [required: >=0.26, installed: 0.35.1]
  - tensorflow-estimator [required: >=2.4.0rc0,<2.5.0, installed: 2.4.0rc0]
  - termcolor [required: ~=1.1.0, installed: 1.1.0]
  - typing-extensions [required: ~=3.7.4, installed: 3.7.4.3]
  - wheel [required: ~=0.35, installed: 0.35.1]
  - wrapt [required: ~=1.12.1, installed: 1.12.1]
tensorflow-addons==0.11.2
  - typeguard [required: >=2.7, installed: 2.10.0]
tf-explain==0.2.1
  - opencv-python [required: >=4.1.0.25, installed: 4.4.0.46]
    - numpy [required: >=1.17.3, installed: 1.19.4]
tf-keras-vis==0.5.3
  - imageio [required: Any, installed: 2.9.0]
    - numpy [required: Any, installed: 1.19.4]
    - pillow [required: Any, installed: 8.0.1]
  - numpy [required: Any, installed: 1.19.4]
  - pillow [required: Any, installed: 8.0.1]
  - scipy [required: Any, installed: 1.5.4]
    - numpy [required: >=1.14.5, installed: 1.19.4]
toolz==0.11.1
xlrd==1.2.0

[bersbersbers]

The above is with pip==20.2.4. I get an error for pip==20.3b1 (which does not reproduce in the current dev version, so I'll skip posting that one). In the current dev version, runtime of the (new default) resolver is down to 16 seconds (compared to 9 seconds with the old resolver in 20.2.4).

[uranusjr]

Sounds like you’re getting the benefit from #9078 (merged after 20.2.4). Closing, feel free to reopen if you don’t think this assessment is correct.

[uranusjr]

The new resolver is always going to be slower, since it actually tries to make sure your requirement set is consistent. You do not need those checks here since all your requirements are pinned exactly with ==, and should be able to skip all the checks with --no-deps.

[uranusjr]

--outdated is slower here also because all the requirements are already installed. Those on-disk installation can be re-used directly; --outdated, by contrast, needs to make additional queries to know whether the installed version is outdated or not, and thus slower.

[pradyunsg]

Overall, everything that you've observed is expected.

There's a (usually small) performance hit when using the new resolver -- which is only used in the install, wheel and download commands BTW. It's a new chunk of code, with different performance characteristics than the old resolver, because this one takes dependency conflicts into account.

[bersbersbers]

The new resolver is always going to be slower, since it actually tries to make sure your requirement set is consistent.

I do understand that in principle. I still wonder what it is actually doing for 45 (now 7, but still) seconds. If it's not network requests and associated delays, I see nothing that should take that long. Then again, I am only an interested user, so there may be good reasons for that, and if you tell me that's expected, I am going to believe you :)

You do not need those checks here since all your requirements are pinned exactly with ==

I'm not sure this is correct: note that I am using sed 's/==.*//' to remove all these pinnings.

and should be able to skip all the checks with --no-deps.

I'm not sure this is correct, either: what if a package is updated to include a new requirement? I certainly want these to be installed, too.

--outdated is slower here also because all the requirements are already installed. Those on-disk installation can be re-used directly; --outdated, by contrast, needs to make additional queries to know whether the installed version is outdated or not, and thus slower.

But the dependency resolvers needs to make these exact same queries, don't they? Note that my xargs pip install is passed all installed packages, and to me, --upgrade-strategy eager implies that they will make "additional queries to know whether the installed version is outdated or not". Again, I don't see why this should be slower. Compatible with this, as noted above, I saw that there were no additional "GET" queries.

[brainwane]

@bersbersbers thank you for your questions! Could I ask you to copy or move your last comment to #9187 if you are still experiencing the problem with pip 20.3? Sorry for the bother.