[RFC] [python] 'setup.py install' is deprecated

[jameslamb]

Description

Running python setup.py install for the Python package raises the following deprecation warning:

/Users/jlamb/mambaforge/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(

I found a very detailed write-up explaining this issue: "Why you shouldn't invoke setup.py directly" (October 2021).

Here are some of the main things we need to know:

...as of the last few years all direct invocations of setup.py are effectively deprecated in favor of invocations via purpose-built and/or standards-based CLI tools like pip, build and tox.

As mentioned in this issue, direct setup.py invocations have effectively been unmaintained for several years now. Whenever someone raises an issue with a reproducer that involves invoking setup.py, the maintainers ask for a reproducer that doesn't hit this code path, and if one can't be found the issue is closed. Put another way, direct invocations of setup.py that currently work for you do so essentially by chance — if something breaks, you are on your own.

Reproducible example

I observed this warning building from source on macOS 12.2.1, with Python v3.9.7 and setuptools==59.8.0.

git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM/python-package

This warning is raised with python setup.py install.

python setup.py install

It is also raised with pip.

pip install -v .

You can also see this warning on LightGBM's CI builds that use python setup.py {command}. For example, the most recent Linux regular job running on Azure DevOps (build link).

References

This warning was added in setuptools=v58.3.0 (October 22, 2021), pypa/setuptools#2824.

That PR mentions https://www.python.org/dev/peps/pep-0517/ as a description of the standard packages should meet to continue to be compliant with the main Python package management tools.

The warning comes from setuptools.command.install.initialize_options(): https://github.com/pypa/setuptools/blob/391bb5d4d09c9eb8d6b2b98968e623455ae0a384/setuptools/command/install.py#L32-L38.

lightgbm currently sub-classes setuptools.command.install to customize the behavior of setup.py install.

LightGBM/python-package/setup.py

Lines 220 to 225 in f6d654b

	class CustomInstall(install):
	user_options = install.user_options + LIGHTGBM_OPTIONS
	def initialize_options(self) -> None:
	install.initialize_options(self)

It also does that to customize setup.py {install_lib, bdist_wheel, sdist}.

LightGBM/python-package/setup.py

Lines 361 to 366 in f6d654b

	cmdclass={
	'install': CustomInstall,
	'install_lib': CustomInstallLib,
	'bdist_wheel': CustomBdistWheel,
	'sdist': CustomSdist,
	},

• "pyproject.toml quickstart" (setuptools docs)
• "Configuring setuptools using setup.cfg files" (setuptools docs)

Why does this matter?

If nothing is changed, the LightGBM Python package might eventually not be installable with some future versions of Python / setuptools.

What should be done?

I don't have a recommendation yet.

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html, https://www.python.org/dev/peps/pep-0517/, and the other things they link to are very detailed and I'm not deeply familiar with the details of how setuptools works.

Just opening this up to track discussion about this issue.

[StrikerRUS]

Oh, it's a rabbit hole!
The most suffered projects are ones that use Cython and numpy to build their source code. I learned that during working on scikit-learn-contrib/lightning#188.

Linking some more related discussions:

• how to handle numpy.distutils and setuptools interaction pypa/setuptools#2372
• RFC Migrate away from numpy.distutils before CPython 3.12 (October 2023) scikit-learn/scikit-learn#21499
• RFC: switch to Meson as a build system scipy/scipy#13615
• https://iscinumpy.gitlab.io/post/scikit-build-proposal/

TL;DR: the future of Python building system is extremely foggy.

[jameslamb]

Thanks for those links @StrikerRUS !

I think scikit-build is really compelling, since it supports CMake. I'll add two more relevant projects here:

• https://pybind11.readthedocs.io/en/stable/
• https://cibuildwheel.readthedocs.io/en/stable/

[StrikerRUS]

Recently released scipy 1.9.0 is the first version built with Meson: http://scipy.github.io/devdocs/release.1.9.0.html#scipy-switched-to-meson-as-its-build-system.

[edzul]

Hi,

I tried with pip as the warning suggests as follows:

Instead of

python setup.py install --precompile

I ran, in the LightGBM/python-package folder

python -m pip install --install-option=--precompile --user .

The --user param is probably not necessary in all cases, but on the machine I'm working is required since site-packages folder is at a specific location specified in the USER_SITE variable. You can try without it, but if you run pip list and don't see lightgbm, try adding the --user param.

And with this there is no warning and works fine.

[ifredom]

it's work for me

pip install setuptools==58.2.0

[jameslamb]

Thank you, but "just pin to a 1+-year-old version of setuptools" is not an acceptable constraint for LightGBM to put on its users.

[jameslamb]

python setup.py install now raises a stricter, slightly more worrying deprecation warning.

DEPRECATION: lightgbm is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currently disables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at pypa/pip#11451

pip is currently on 22.3.1 (release history), so I'm not sure when the 23.x series will start.

But I think this adds new urgency to the project to switch lightgbm over to a pyproject.toml-based build.

If no one else picks it up, I'll try working on that after resolving #5514 .

[jameslamb]

Adding more deprecation warnings noticed in one of the CUDA builds today (build link).

WARNING: Implying --no-binary=:all: due to the presence of --build-option / --global-option / --install-option. Consider using --config-settings for more flexibility.

DEPRECATION: --install-option is deprecated because it forces pip to use the 'setup.py install' command which is itself deprecated. pip 23.1 will enforce this behaviour change. A possible replacement is to use --config-settings. Discussion can be found at pypa/pip#11358

DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at pypa/pip#11453

[jameslamb]

Adding that there is also a PEP 517 backend for MSBuild:

• https://pypi.org/project/pymsbuild/
• https://github.com/zooba/pymsbuild

LightGBM's setup.py currently supports MSBuild on Windows as an alternative to CMake.

LightGBM/python-package/setup.py

Lines 174 to 188 in d067ab6

	if not any((use_gpu, use_cuda, use_cuda_exp, use_mpi, use_hdfs, nomp, bit32, integrated_opencl)):
	logger.info("Starting to compile with MSBuild from existing solution file.")
	platform_toolsets = ("v143", "v142", "v141", "v140")
	for pt in platform_toolsets:
	status = silent_call(["MSBuild",
	str(CURRENT_DIR / "compile" / "windows" / "LightGBM.sln"),
	"/p:Configuration=DLL",
	"/p:Platform=x64",
	f"/p:PlatformToolset={pt}"])
	if status == 0 and lib_path.is_file():
	break
	else:
	clear_path(CURRENT_DIR / "compile" / "windows" / "x64")
	if status != 0 or not lib_path.is_file():
	logger.warning("Compilation with MSBuild from existing solution file failed.")

[jameslamb]

Assigning this to myself. I've started working on it on my fork.

[jameslamb]

adding a comment to collect some relevant links:

• https://snarky.ca/the-challenges-in-designing-a-library-for-pep-425/
• https://github.com/RalfG/python-wheels-manylinux-build
• https://github.com/scikit-build/scikit-build-sample-projects/tree/master/projects/hello-cmake-package

[jameslamb]

pip just removed the behavior "fall back to running python setup.py install" upstream: pypa/pip#11874

I believe that means that as of the next pip release, lightgbm source distributions will no longer be installable with pip install and users on platforms we don't provide wheels for will have to pin to older versions of pip.

I'll prioritize #5759 this week...on that PR, I have a working setup for building source distributions that don't require setup.py. I just need to get it working with two more things:

• forcing the use of MinGW
• passing through stuff like OpenCL and Boost directories

Will @ when it's ready for review. Worst case scenario, we can move forward with that PR even without "force MinGW" and "customize GPU build" support for now, since I expect those options are used by a small minority of LightGBM users.

[jameslamb]

pip 23.1 was just released, removing support for --install-option.

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --install-option

That release also deprecated --build-option and --global-option in favor of passing things to build backends via --config-setting.

https://pip.pypa.io/en/stable/news/

This adds new urgency to this work. I'll continue with #5759 this week and hopefully have it ready for review soon.

[jameslamb]

Also want to add that xgboost is currently working on removing their setup.py: dmlc/xgboost#9021

[jameslamb]

For those following along here, we just took an important step... LightGBM is no longer support running python setup.py {command} in this repo.

Instead, to customize installation of the Python package from a clone of this git repo, we now encourage the use of a shell script run from the root of the repo.

For example:

sh ./build-python.sh install --cuda

The next release of LightGBM will also include a source distribution published to PyPI which can be passed similar customization flags via pip install (no need to clone the repo), just as you can do today with something like pip install :no-binary: lightgbm --install-option=--cuda.

---

In future PRs, LightGBM's setup.py will be removed entirely, in favor of a PEP 517/518 compliant build backend (most likely scikit-build-core).

[jameslamb]

The warnings from Python build tools about not using setup.py are getting louder and more urgent.

  INFO:root:running bdist_wheel
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-qy0gt00m/setup.py:195: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    install.initialize_options(self)

(build link)

[mirekphd]

python -m pip install --install-option=--precompile --user .

Please note that this workaround no longer works, since the deprecated argument --install-option has been removed completely from pip==23.1 (see docs).

I believe the only option left is to use the build script from the LightGBM repo:
[ba]sh ./build-python.sh install --precompile

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.