For some packages that use PEP-517 and setuptools, getting wheel metadata involves building the package. How to prevent it?

[tvalentyn]

A project I am working on is considering using PEP-517. We also currently use setuptools. I have observed that after numpy switched to use PEP-517 in numpy/numpy#14053,
there was a regression in pip download --dest /tmp numpy==1.21.2 --no-deps --no-binary :all: behavior. When a user runs the command, pip ends up building the package (which is slow), see: numpy/numpy#14053 (comment).

The explanation is that after downloading an sdist, pip needs to retrieve and verify package metadata . If a project uses PEP-517, then pip makes a call to prepare_metadata_for_build_wheel hook on the package's build backend (in our case, setuptools).

It appears that in numpy+setuptools case, this call is inefficient and might involve building entire wheel. Do we have an issue in setuptools' implementation of prepare_metadata_for_build_wheel? FWIW, python setup.py egg_info and python setup.py dist_info finish fast on an unpacked numpy sdist.

If it is a numpy build configuration issue, how could it be prevented? Also, is there a way I can invoke setuptools' prepare_metadata_for_build_wheel hook on my project's codebase to see how long it would take and avoid this inefficiency?

Thank you.

[tvalentyn]

pip's verbose mode provides clues as to what is happening. If you run:

python3 -m pip -v download --dest /tmp numpy==1.21.2 --no-deps --no-binary  :all:

we'll see that most of the time is spent on getting project's build dependencies.

Collecting numpy==1.21.2
  File was already downloaded /tmp/numpy-1.21.2.zip
  Running command /home/valentyn/.pyenv/versions/py373/bin/python3 /tmp/pip-standalone-pip-p49e98p9/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-uv05j1h6/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'packaging==20.5; platform_machine=='"'"'arm64'"'"'' 'setuptools<49.2.0' wheel==0.36.2 'Cython>=0.29.24,<3.0'
  Ignoring packaging: markers 'platform_machine == "arm64"' don't match your environment
  Collecting setuptools<49.2.0
    Using cached setuptools-49.1.3.zip (2.2 MB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Collecting wheel==0.36.2
    Using cached wheel-0.36.2.tar.gz (65 kB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Collecting Cython<3.0,>=0.29.24
    Using cached Cython-0.29.24.tar.gz (2.1 MB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Skipping wheel build for wheel, due to binaries being disabled for it.
  Skipping wheel build for Cython, due to binaries being disabled for it.
  Skipping wheel build for setuptools, due to binaries being disabled for it.
  Installing collected packages: wheel, setuptools, Cython
      Running setup.py install for wheel: started
      Running setup.py install for wheel: finished with status 'done'
      Running setup.py install for setuptools: started
      Running setup.py install for setuptools: finished with status 'done'
      Running setup.py install for Cython: started
      Running setup.py install for Cython: finished with status 'done'
  Successfully installed Cython-0.29.24 setuptools-49.1.3 wheel-0.36.2
  Installing build dependencies ... done
  Running command /home/valentyn/.pyenv/versions/py373/bin/python3 /home/valentyn/.pyenv/versions/py373/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpi20a_gyu
  running egg_info
  running build_src
  build_src
  creating numpy.egg-info
  writing numpy.egg-info/PKG-INFO
  writing dependency_links to numpy.egg-info/dependency_links.txt
  writing entry points to numpy.egg-info/entry_points.txt
  writing top-level names to numpy.egg-info/top_level.txt
  writing manifest file 'numpy.egg-info/SOURCES.txt'
  reading manifest file 'numpy.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'LICENSES_bundled.txt'
  warning: no previously-included files found matching 'azure-*.yml'
  no previously-included directories found matching 'doc/build'
  no previously-included directories found matching 'doc/source/generated'
  no previously-included directories found matching 'benchmarks/env'
  no previously-included directories found matching 'benchmarks/results'
  no previously-included directories found matching 'benchmarks/html'
  no previously-included directories found matching 'benchmarks/numpy'
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '*.pyd' found anywhere in distribution
  warning: no previously-included files matching '*.swp' found anywhere in distribution
  warning: no previously-included files matching '*.bak' found anywhere in distribution
  warning: no previously-included files matching '*~' found anywhere in distribution
  writing manifest file 'numpy.egg-info/SOURCES.txt'
  Running from numpy source directory.
  Getting requirements to build wheel ... done

and, since pip command says --no-binary :all:, during metadata generation step, we install all package dependencies from sources into pip's isolated build environment, note the lines Skipping wheel build for wheel, due to binaries being disabled for it.

The actual metadata generation takes a ~second or less and happens here:

Running command /home/valentyn/.pyenv/versions/py373/bin/python3 /home/valentyn/.pyenv/versions/py373/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkvbe69q1
  running dist_info
  running build_src
  build_src
  creating /tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info
  writing /tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/entry_points.txt
  writing top-level names to /tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'LICENSES_bundled.txt'
  warning: no previously-included files found matching 'azure-*.yml'
  no previously-included directories found matching 'doc/build'
  no previously-included directories found matching 'doc/source/generated'
  no previously-included directories found matching 'benchmarks/env'
  no previously-included directories found matching 'benchmarks/results'
  no previously-included directories found matching 'benchmarks/html'
  no previously-included directories found matching 'benchmarks/numpy'
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '*.pyd' found anywhere in distribution
  warning: no previously-included files matching '*.swp' found anywhere in distribution
  warning: no previously-included files matching '*.bak' found anywhere in distribution
  warning: no previously-included files matching '*~' found anywhere in distribution
  writing manifest file '/tmp/pip-modern-metadata-dfhji1uj/numpy.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-dfhji1uj/numpy.dist-info'
  adding license file "LICENSE.txt" (matched pattern "LICEN[CS]E*")
  Running from numpy source directory.
  Preparing wheel metadata (pyproject.toml) ... done
Successfully downloaded numpy

[webknjaz]

FYI --no-binary your-project-name is a thing.

[tvalentyn]

Sure, however I have another use-case where I also want sources of all dependencies for my packages, and will omit --no-deps, then command is like: python3 -m pip -v download --dest /tmp numpy==1.21.2 --no-binary :all:, and I can no longer use --no-binary project-name.
I think the inefficiency here is that pip is applying --no-binary to the scenario where pip requests prepare_metadata_for_build_wheel. It is during this step that pip creates an isolated build environment, solely for the purposes of getting said metadata, and then it installs projects build dependencies into the isolated environment from sources. As a user, I don't have a preference whether pip creates isolated build environment from sdists or bdists, I only need sources that I requested (and by the time metadata is checked, sources have already been fetched).

[tvalentyn]

If it is a numpy build configuration issue, how could it be prevented? Also, is there a way I can invoke setuptools' prepare_metadata_for_build_wheel hook on my project's codebase to see how long it would take and avoid this inefficiency?

To answer my own question, I think it will take as much time as it will take to download and install sdists for build dependencies of a package.

Opened pypa/pip#10589 to further discuss this.