v0.4.14 seems to have resolver issues

[stas00]

our uv pip compile -v setup.py --extra replug has just started failing on CI once uv has been updated:

  × No solution found when resolving dependencies:
  ╰─▶ Because nmslib==2.1.1 depends on numpy>=1.10.0,<1.17 and only nmslib<=2.1.1 is available, we can conclude that nmslib>=2.1.1
      depends on numpy>=1.10.0,<1.17.
      And because pyserini==0.23.0 depends on numpy>=1.18.1, we can conclude that nmslib>=2.1.1 and pyserini==0.23.0 are incompatible.
      And because pyserini==0.23.0 depends on nmslib>=2.1.1 and dawn depends on pyserini==0.23.0, we can conclude that your requirements
      are unsatisfiable.

The parser seems to be getting the requirements wrong, because:

$ git clone https://github.com/nmslib
$ cd nmslib
$ git checkout v2.1.1
$ grep -Ir "<1\.17"
python_bindings/setup.py:dep_list.append("numpy>=1.10.0,<1.17 ; python_version=='2.7'")

but I'm running py3.10

an easier way to repro is to just run:

$ uv pip install nmslib==2.1.1
Resolved 4 packages in 152ms
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: numpy==1.16.6
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:

--- stderr:
Running from numpy source directory.
/data/env/cache/uv/sdists-v4/pypi/numpy/1.16.6/YYsNUdkR_MaadjuXSLKgQ/numpy-1.16.6.zip/numpy/distutils/misc_util.py:476: SyntaxWarning: "is" with a literal. Did you mean "=="?
  return is_string(s) and ('*' in s or '?' is s)
Traceback (most recent call last):
  File "<string>", line 11, in <module>
  File "/env/cache/uv/builds-v0/.tmpEg6RuR/lib/python3.10/site-packages/setuptools/build_meta.py", line 421, in build_wheel
    return self._build_with_temp_dir(
  File "/env/cache/uv/builds-v0/.tmpEg6RuR/lib/python3.10/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
    self.run_setup()
  File "/env/cache/uv/builds-v0/.tmpEg6RuR/lib/python3.10/site-packages/setuptools/build_meta.py", line 503, in run_setup
    super().run_setup(setup_script=setup_script)
  File "/env/cache/uv/builds-v0/.tmpEg6RuR/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 419, in <module>
  File "<string>", line 398, in setup_package
  File "/data/env/cache/uv/sdists-v4/pypi/numpy/1.16.6/YYsNUdkR_MaadjuXSLKgQ/numpy-1.16.6.zip/numpy/distutils/core.py", line 26, in <module>
    from numpy.distutils.command import config, config_compiler, \
  File "/data/env/cache/uv/sdists-v4/pypi/numpy/1.16.6/YYsNUdkR_MaadjuXSLKgQ/numpy-1.16.6.zip/numpy/distutils/command/config.py", line 19, in <module>
    from numpy.distutils.mingw32ccompiler import generate_manifest
  File "/data/env/cache/uv/sdists-v4/pypi/numpy/1.16.6/YYsNUdkR_MaadjuXSLKgQ/numpy-1.16.6.zip/numpy/distutils/mingw32ccompiler.py", line 34, in <module>
    from distutils.msvccompiler import get_build_version as get_build_msvc_version
ModuleNotFoundError: No module named 'distutils.msvccompiler'
---

[stas00]

0.4.13 also has an issue

uv pip install uv==0.4.13
uv pip install nmslib==2.1.1

same error as the end of the OP

I didn't check them all but uv==0.4.0 works fine.

The correct resolution should be numpy>=1.10.0 for py3.10 for nmslib==2.1.1

$ uv pip install uv==0.4.0
Resolved 1 package in 122ms
Prepared 1 package in 396ms
Uninstalled 1 package in 51ms
Installed 1 package in 125ms
 - uv==0.4.13
 + uv==0.4.0

$ uv pip install  nmslib==2.1.1
Resolved 4 packages in 322ms
Installed 1 package in 1.05s
 + nmslib==2.1.1

[stas00]

The actual dependency I need to satisfy is pyserini==0.23.0, which pulls in nmslib.

[zanieb]

Sorry struggling to reproduce this, am I doing something wrong?

❯ echo "nmslib==2.1.1" | uvx [email protected] pip compile - --python-platform linux --python 3.10
Resolved 4 packages in 3ms
# This file was autogenerated by uv via the following command:
#    uv pip compile - --python-platform linux --python 3.10
nmslib==2.1.1
numpy==2.1.1
    # via nmslib
psutil==6.0.0
    # via nmslib
pybind11==2.6.1
    # via nmslib

[stas00]

indeed, seems to be fine alone:

$ echo "nmslib==2.1.1" | uv pip compile - --python-platform linux --python 3.10
Resolved 4 packages in 245ms
# This file was autogenerated by uv via the following command:
#    uv pip compile - --python-platform linux --python 3.10
nmslib==2.1.1
numpy==1.16.6
    # via nmslib
psutil==6.0.0
    # via nmslib
pybind11==2.6.1
    # via nmslib

now let's try with the actual package that I depend on:

$ echo "pyserini==0.23.0" | uv pip compile - --python-platform linux --python 3.10
  × No solution found when resolving dependencies:
  ╰─▶ Because nmslib==2.1.1 depends on numpy>=1.10.0,<1.17 and only nmslib<=2.1.1 is available, we can conclude that nmslib>=2.1.1 depends
      on numpy>=1.10.0,<1.17.
      And because pyserini==0.23.0 depends on numpy>=1.18.1, we can conclude that nmslib>=2.1.1 and pyserini==0.23.0 are incompatible.
      And because pyserini==0.23.0 depends on nmslib>=2.1.1 and you require pyserini==0.23.0, we can conclude that your requirements are
      unsatisfiable.

that numpy>=1.10.0,<1.17 dependency is the trigger, and it should be numpy>=1.10.0 instead.

By looking at the source code of these repos - the only suspect to where <1.17 may have come from I think is:

$ git clone https://github.com/nmslib
$ cd nmslib
$ git checkout v2.1.1
$ grep -Ir "<1\.17"
python_bindings/setup.py:dep_list.append("numpy>=1.10.0,<1.17 ; python_version=='2.7'")

[zanieb]

Looks like this regressed in 0.4.13

❯ echo "pyserini==0.23.0" | uvx [email protected] pip compile - --python-platform linux --python 3.10
Installed 1 package in 6ms
  × No solution found when resolving dependencies:
  ╰─▶ Because nmslib==2.1.1 depends on numpy>=1.10.0,<1.17 and only nmslib<=2.1.1 is available, we can conclude that nmslib>=2.1.1 depends on
      numpy>=1.10.0,<1.17.
      And because pyserini==0.23.0 depends on numpy>=1.18.1, we can conclude that nmslib>=2.1.1 and pyserini==0.23.0 are incompatible.
      And because pyserini==0.23.0 depends on nmslib>=2.1.1 and you require pyserini==0.23.0, we can conclude that your requirements are unsatisfiable.
      
❯ echo "pyserini==0.23.0" | uvx [email protected] pip compile - --python-platform linux --python 3.10
Resolved 83 packages in 27ms

[zanieb]

Unfortunately this was caused by 18c18b8 (#7556) which we thought just affected error messages. Reverting the commit causes resolution to succeed.

cc @charliermarsh

[charliermarsh]

I'll do a release real-quick to get this revert out.

[charliermarsh]

Okay, I think the root problem here is that nmslib has inconsistent metadata across wheels... So some of the wheels contain:

Requires-Dist: numpy (<1.17,>=1.10.0) ; python_version == "2.7"
Requires-Dist: numpy (>=1.10.0) ; python_version >= "3.5"

And we succeed in that case.

Others contain this definition, which looks like a mistake:

Requires-Dist: numpy (<1.17,>=1.10.0)
Requires-Dist: numpy (>=1.10.0)

If we can't find a valid wheel for a dependency, we'll still read metadata from an incompatible wheel. The change I made was actually fine, it that it started preferring different incompatible wheels than before (wheels that are closer to being compatible). But I guess we were just lucky before that we got the "good" metadata vs. the "bad" metadata here.

[charliermarsh]

I created an issue here: nmslib/nmslib#558

[stas00]

Thank you for getting to the root of it, Charlie.

Yes, their last released package is many years old. I'm asking already to get a new source release made, but it appears to be far from trivial according to the owner. Not sure what to do. castorini/pyserini#1985

Thank you for flagging this issue to the owner of that package, Charlie.

[charliermarsh]

No problem. The other thing we could do is curate a list of "rejected" wheels in uv itself... That feels wrong and I'd prefer to avoid it, but we may be forced into it eventually.

[stas00]

That would require a lot of unnecessary maintenance work on your part. I wonder if uv falling back to pip on failure would be a sufficient workaround. Like having a new flag, --fallback-to-pip-on-failure, which would require pip to be installed (assuming pip has a more relaxed/less precise resolver) e.g. often when using conda I had to fall back to pip because conda was super anal about some dependency situations and pip saved the day and everything worked just fine.)