Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use uv for installing requirements dynamically in stubtest_third_party.py #11531

Closed
wants to merge 4 commits into from

Conversation

AlexWaygood
Copy link
Member

Follow-up to #11517

@JelleZijlstra
Copy link
Member

Seems like passlib has an undeclared runtime dependency on pkg_resources, fun.

@AlexWaygood AlexWaygood marked this pull request as draft March 4, 2024 17:55
@AlexWaygood
Copy link
Member Author

AlexWaygood commented Mar 4, 2024

Seems like passlib has an undeclared runtime dependency on pkg_resources, fun.

Ouch, I'm surprised they haven't fixed that yet since Python 3.12 doesn't install setuptools into venvs by default anymore?

@AlexWaygood
Copy link
Member Author

the playsound error looks much more fun:

Failed to install
error: Failed to download and build: playsound==1.3.0
  Caused by: Failed to build: playsound==1.3.0
  Caused by: Build backend failed to determine extra requires with `build_wheel()`:
--- stdout:

--- stderr:
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/Users/alexw/Library/Caches/uv/.tmpRKpAih/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexw/Library/Caches/uv/.tmpRKpAih/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/Users/alexw/Library/Caches/uv/.tmpRKpAih/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 487, in run_setup
    super().run_setup(setup_script=setup_script)
  File "/Users/alexw/Library/Caches/uv/.tmpRKpAih/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 6, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py", line 1282, in getsource
    lines, lnum = getsourcelines(object)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py", line 1264, in getsourcelines
    lines, lnum = findsource(object)
                  ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py", line 1093, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

@AlexWaygood
Copy link
Member Author

Roundup of the failures

Build-time failures (not solvable without changes to the upstream packages and/or uv)

Runtime failures (these can be solved by installing setuptools into the environment before each package's stubtest run, like I did with my latest commit):

  • libsass:
    • The sassutils package imports setuptools and pkg_resources (which is distributed with setuptools) without declaring a dependency on setuptools. setuptools is automatically installed into a virtual environment on Python <3.12 with the stdlib venv module, and you can get similar behaviour with uv if you specify --seed (but uv, like the stdlib venv module, doesn't include setuptools by default on py312+, even if you specify --seed).
    • I wondered whether sassutils was meant to be public API or not, since the main package distributed with libsass is the sass package. But sassutils does seem to be documented.
  • passlib imports from pkg_resources without declaring a dependency on setuptools.
  • google-cloud-ndb:

@srittau
Copy link
Collaborator

srittau commented Mar 5, 2024

While the performance improvements would obviously be nice, this doesn't sound as if uv was really production ready, at least not for our purposes. I'd prefer not to add this additional headache at this point.

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Mar 5, 2024

While the performance improvements would obviously be nice, this doesn't sound as if uv was really production ready, at least not for our purposes. I'd prefer not to add this additional headache at this point.

Strictly speaking, these are all errors in third-party packages rather than uv, per se (except for possibly playsound?). But, fully agreed that this would cause unnecessary complication for our purposes right now.

@AlexWaygood AlexWaygood closed this Mar 5, 2024
@Avasam
Copy link
Collaborator

Avasam commented Mar 6, 2024

If anything, it at least revealed some issues in the ecosystem that can be raised upstream. I can't blame fasttext for assuming that pip is always installed, for example, but Python packaging is changing and some assumptions can't be made anymore.

@charliermarsh
Copy link

👍 The only thing I'd add is that if packages fail to install with uv in these ways, it tends to be the case that they also fail to install with pip when using --use-pep517. So the problems with these packages aren't uv-specific per se.

For example, here's dateparser[fasttext] with the same No module named pip at the top:

❯ pip install -e ".[fasttext]" --use-pep517
Obtaining file:///Users/crmarsh/workspace/puffin/dateparser
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: python-dateutil in /Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages (from dateparser==1.2.0) (2.9.0.post0)
Requirement already satisfied: pytz in /Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages (from dateparser==1.2.0) (2024.1)
Requirement already satisfied: regex!=2019.02.19,!=2021.8.27 in /Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages (from dateparser==1.2.0) (2023.12.25)
Requirement already satisfied: tzlocal in /Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages (from dateparser==1.2.0) (5.2)
Collecting fasttext (from dateparser==1.2.0)
  Downloading fasttext-0.9.2.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.8/68.8 kB 3.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      /Users/crmarsh/.local/share/rtx/installs/python/3.12.1/bin/python3.12: No module named pip
      Traceback (most recent call last):
        File "<string>", line 38, in __init__
      ModuleNotFoundError: No module named 'pybind11'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/crmarsh/.local/share/rtx/installs/python/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/pip-build-env-c16570gj/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/pip-build-env-c16570gj/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/pip-build-env-c16570gj/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/pip-build-env-c16570gj/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 72, in <module>
        File "<string>", line 41, in __init__
      RuntimeError: pybind11 install failed.
      [end of output]

@srittau
Copy link
Collaborator

srittau commented Mar 7, 2024

One thing we could do is start using pip --use-pep517 as soon as the existing packages are fixed and start rejecting third-party package that are not installable that way. Then we can switch to uv at some later point as soon as the remaining issues are ironed out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants