[python-package] Build from Sources failed

[boscocho]

Description

Trying to build from sources under python-package but got following error message:

(base) PS F:\src\exp\LightGBM\python-package> pip install --no-binary lightgbm lightgbm
Defaulting to user installation because normal site-packages is not writeable
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting lightgbm
  Using cached lightgbm-3.3.5.tar.gz (1.5 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: wheel in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (0.38.4)
Requirement already satisfied: numpy in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.24.3)
Requirement already satisfied: scipy in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.10.1)
Requirement already satisfied: scikit-learn!=0.22.0 in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.2.2)
Requirement already satisfied: joblib>=1.1.1 in c:\programdata\miniconda3\lib\site-packages (from scikit-learn!=0.22.0->lightgbm) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\programdata\miniconda3\lib\site-packages (from scikit-learn!=0.22.0->lightgbm) (3.1.0)
Installing collected packages: lightgbm
  DEPRECATION: lightgbm is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currently disables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/11451
  Running setup.py install for lightgbm ... error
  error: subprocess-exited-with-error

  × Running setup.py install for lightgbm did not run successfully.
  │ exit code: 1
  ╰─> [32 lines of output]
      INFO:root:running install
      C:\ProgramData\miniconda3\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      INFO:LightGBM:Starting to compile the library.
      INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
      WARNING:LightGBM:Compilation with MSBuild from existing solution file failed.
      INFO:LightGBM:Starting to compile with Visual Studio 16 2019 (x64).
      INFO:LightGBM:Starting to compile with Visual Studio 15 2017 (x64).
      INFO:LightGBM:Starting to compile with Visual Studio 14 2015 (x64).
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\ychiu\AppData\Local\Temp\pip-install-1g7ts6p9\lightgbm_3a2cce6a9b294f82832195b7b67412d8\setup.py", line 334, in <module>
          setup(name='lightgbm',
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\ProgramData\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\ychiu\AppData\Local\Temp\pip-install-1g7ts6p9\lightgbm_3a2cce6a9b294f82832195b7b67412d8\setup.py", line 248, in run
          compile_cpp(use_mingw=self.mingw, use_gpu=self.gpu, use_cuda=self.cuda, use_mpi=self.mpi,
        File "C:\Users\ychiu\AppData\Local\Temp\pip-install-1g7ts6p9\lightgbm_3a2cce6a9b294f82832195b7b67412d8\setup.py", line 192, in compile_cpp
          raise Exception("\n".join(('Please install Visual Studio or MS Build and all required dependencies first',
      Exception: Please install Visual Studio or MS Build and all required dependencies first
      The full version of error log was saved into C:\Users\ychiu\LightGBM_compilation.log
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lightgbm

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I checked C:\Users\ychiu\LightGBM_compilation.log:

CMake Error at CMakeLists.txt:32 (PROJECT):
  Generator

    Visual Studio 16 2019

  could not find any instance of Visual Studio.

-- Configuring incomplete, errors occurred!
CMake Error at CMakeLists.txt:32 (PROJECT):
  Generator

    Visual Studio 15 2017

  could not find any instance of Visual Studio.

-- Configuring incomplete, errors occurred!
-- Selecting Windows SDK version  to target Windows 10.0.19045.
CMake Error at CMakeLists.txt:32 (PROJECT):
  Failed to run MSBuild command:

    MSBuild.exe

  to get the value of VCTargetsPath:

    The system cannot find the file specified

-- Configuring incomplete, errors occurred!

On my computer, I have Visual Studio 2022 & cmake 3.26.4 installed,
also I have made sure cmake is in system PATH that I was able to following instructions on windows build (not python-package) without problem.

Reproducible example

pip install --no-binary lightgbm lightgbm

Environment info

LightGBM version or commit hash:

The latest one I got by using:

git clone --recursive https://github.com/microsoft/LightGBM

Command(s) you used to install LightGBM

git clone --recursive https://github.com/microsoft/LightGBM

I checked the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0

originally there is no VCTargetsPath there and I manually added VCTargetsPath with value $(MSBuildExtensionsPath32)\Microsoft.Cpp\v4.0\v140\

but still got the same error message :(

Additional Comments

[jameslamb]

Thanks for using LightGBM.

To start, I re-formatted the text of your issue a bit, to more clearly separate logs and code from your own statements. Please see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax for some advice on how to format text on GitHub.

---

I have some questions.

1. Are you trying to build the lightgbm Python package from the source code on GitHub, or to install the latest release hosted on PyPI?

You said you ran pip install lightgbm (which installs from PyPI) but also git clone --recursive https://github.com/microsoft/LightGBM (which copies the source code from GitHub).

2. What happens when you run the following commands from the root of the repository?

rm -f ./build
mkdir build
cd build
cmake ..
make -j2

[boscocho]

Thanks James for your response :)

I was trying to build the lightgbm python package from the source code which I cloned from GitHub.
Because I got error message when I ran

pip install --no-binary lightgbm lightgbm

I tried to lightgbm windows build to verify if my cmake & visual studio 2022 have any issues.

2. Here is the output I got:

MINGW64 /f/src/exp/LightGBM/build (master)
$ cmake ..
\-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.19045.
-- The C compiler identification is MSVC 19.36.32532.0
-- The CXX compiler identification is MSVC 19.36.32532.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Failed
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Configuring done (16.6s)
-- Generating done (0.1s)
-- Build files have been written to: F:/src/exp/LightGBM/build

 MINGW64 /f/src/exp/LightGBM/build (master)
$ make -j2
make: *** No targets specified and no makefile found.  Stop.

seems cmake succeeded but make failed :(
it was good when I used:

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake -A x64 -DUSE_OPENMP=OFF ..
cmake --build . --target ALL_BUILD --config Release

or

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake -A x64 -DUSE_MPI=ON ..
cmake --build . --target ALL_BUILD --config Release

[jameslamb]

make: *** No targets specified and no makefile found. Stop.

You missed running cd build before make -j2.

trying to build from sources in GitHub

Ok, please try running the following from the root of the repository.

sh build-python.sh install

Don't run pip install lightgbm if your goal is to build from source in this repo. That command downloads the source code from PyPI.

[boscocho]

Thank you James!
I was able to run

sh build-python.sh install

without problem.
One problem I have is that when I tried to verify if I did things correctly, I changed the warning text on io\dataset.cpp

"Auto-choosing col-wise multi-threading, the overhead of testing was "

to

"**ABC** Auto-choosing col-wise multi-threading, the overhead of testing was "

and run

sh build-python.sh install

with output log:

(base) PS F:\src\exp\LightGBM> sh build-python.sh install
building lightgbm
--- building wheel ---
INFO:root:running bdist_wheel
C:\ProgramData\miniconda3\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
INFO:root:running build
INFO:root:running build_py
INFO:root:creating build
INFO:root:creating build\lib
INFO:root:creating build\lib\lightgbm
INFO:root:copying lightgbm\basic.py -> build\lib\lightgbm
INFO:root:copying lightgbm\callback.py -> build\lib\lightgbm
INFO:root:copying lightgbm\compat.py -> build\lib\lightgbm
INFO:root:copying lightgbm\dask.py -> build\lib\lightgbm
INFO:root:copying lightgbm\engine.py -> build\lib\lightgbm
INFO:root:copying lightgbm\libpath.py -> build\lib\lightgbm
INFO:root:copying lightgbm\plotting.py -> build\lib\lightgbm
INFO:root:copying lightgbm\sklearn.py -> build\lib\lightgbm
INFO:root:copying lightgbm\__init__.py -> build\lib\lightgbm
INFO:root:running egg_info
INFO:root:creating lightgbm.egg-info
INFO:root:writing lightgbm.egg-info\PKG-INFO
INFO:root:writing dependency_links to lightgbm.egg-info\dependency_links.txt
INFO:root:writing requirements to lightgbm.egg-info\requires.txt
INFO:root:writing top-level names to lightgbm.egg-info\top_level.txt
INFO:root:writing manifest file 'lightgbm.egg-info\SOURCES.txt'
INFO:root:reading manifest file 'lightgbm.egg-info\SOURCES.txt'
INFO:root:reading manifest template 'MANIFEST.in'
WARNING:root:no previously-included directories found matching 'build'
WARNING:root:warning: no files found matching '*.txt'
WARNING:root:warning: no files found matching '*.so' under directory 'lightgbm'
WARNING:root:warning: no files found matching '*.so' under directory 'compile'
WARNING:root:warning: no files found matching '*.dll' under directory 'compile\Release'
WARNING:root:warning: no files found matching '*.dll' under directory 'compile\windows\x64\DLL'
WARNING:root:warning: no previously-included files matching '*.py[co]' found anywhere in distribution
WARNING:root:warning: no previously-included files found matching 'compile\external_libs\compute\.git'
INFO:root:adding license file 'LICENSE'
INFO:root:writing manifest file 'lightgbm.egg-info\SOURCES.txt'
INFO:root:copying lightgbm\VERSION.txt -> build\lib\lightgbm
INFO:root:copying lightgbm\py.typed -> build\lib\lightgbm
INFO:wheel:installing to build\bdist.win-amd64\wheel
INFO:root:running install
INFO:LightGBM:Starting to compile the library.
INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
WARNING:LightGBM:Compilation with MSBuild from existing solution file failed.
INFO:LightGBM:Starting to compile with Visual Studio 17 2022 (x64).
INFO:root:running install_lib
INFO:root:creating build\bdist.win-amd64
INFO:root:creating build\bdist.win-amd64\wheel
INFO:root:creating build\bdist.win-amd64\wheel\lightgbm
INFO:root:copying build\lib\lightgbm\basic.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\callback.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\compat.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\dask.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\engine.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\libpath.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\plotting.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\py.typed -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\sklearn.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\VERSION.txt -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:root:copying build\lib\lightgbm\__init__.py -> build\bdist.win-amd64\wheel\.\lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['F:\\src\\exp\\LightGBM\\lightgbm-python\\compile\\Release\\lib_lightgbm.dll', 'F:\\src\\exp\\LightGBM\\Release\\lib_lightgbm.dll']
INFO:root:copying F:\src\exp\LightGBM\lightgbm-python\compile\Release\lib_lightgbm.dll -> build\bdist.win-amd64\wheel\lightgbm
INFO:root:running install_egg_info
INFO:root:Copying lightgbm.egg-info to build\bdist.win-amd64\wheel\.\lightgbm-3.3.5.99-py3.10.egg-info
INFO:root:running install_scripts
INFO:wheel:creating build\bdist.win-amd64\wheel\lightgbm-3.3.5.99.dist-info\WHEEL
INFO:wheel:creating '../dist\lightgbm-3.3.5.99-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
INFO:wheel:adding 'lightgbm/VERSION.txt'
INFO:wheel:adding 'lightgbm/__init__.py'
INFO:wheel:adding 'lightgbm/basic.py'
INFO:wheel:adding 'lightgbm/callback.py'
INFO:wheel:adding 'lightgbm/compat.py'
INFO:wheel:adding 'lightgbm/dask.py'
INFO:wheel:adding 'lightgbm/engine.py'
INFO:wheel:adding 'lightgbm/lib_lightgbm.dll'
INFO:wheel:adding 'lightgbm/libpath.py'
INFO:wheel:adding 'lightgbm/plotting.py'
INFO:wheel:adding 'lightgbm/py.typed'
INFO:wheel:adding 'lightgbm/sklearn.py'
INFO:wheel:adding 'lightgbm-3.3.5.99.dist-info/LICENSE'
INFO:wheel:adding 'lightgbm-3.3.5.99.dist-info/METADATA'
INFO:wheel:adding 'lightgbm-3.3.5.99.dist-info/WHEEL'
INFO:wheel:adding 'lightgbm-3.3.5.99.dist-info/top_level.txt'
INFO:wheel:adding 'lightgbm-3.3.5.99.dist-info/RECORD'
INFO:wheel:removing build\bdist.win-amd64\wheel
--- installing lightgbm ---
Defaulting to user installation because normal site-packages is not writeable
Looking in links: .
Requirement already satisfied: lightgbm in c:\users\ychiu\appdata\roaming\python\python310\site-packages (3.3.5.99)
Requirement already satisfied: wheel in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (0.38.4)
Requirement already satisfied: scikit-learn!=0.22.0 in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.2.2)
Requirement already satisfied: scipy in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.10.1)
Requirement already satisfied: numpy in c:\programdata\miniconda3\lib\site-packages (from lightgbm) (1.24.3)
Requirement already satisfied: joblib>=1.1.1 in c:\programdata\miniconda3\lib\site-packages (from scikit-learn!=0.22.0->lightgbm) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\programdata\miniconda3\lib\site-packages (from scikit-learn!=0.22.0->lightgbm) (3.1.0)
cleaning up

everything looks successful but when I tested it with simple python example:

(base) PS F:\src\exp\LightGBM> python .\examples\python-guide\simple_example.py
Loading data...
Starting training...
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.002023 seconds.
You can set `force_col_wise=true` to remove the overhead.
Training until validation scores don't improve for 5 rounds
Did not meet early stopping. Best iteration is:
[20]    valid_0's l1: 0.431129  valid_0's l2: 0.198063
Saving model...
Starting predicting...
The RMSE of prediction is: 0.4450426449744025

I still get old message instead of the new one I updated:

[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.002023 seconds.

May I ask if I missed anything in my build?

[boscocho]

found the root cause: Once installed, running build-python.sh script again won't install new light gbm version.
Need to

pip uninstall lightgbm

to remove original install the run build-python.sh again

Suggestion: maybe add

pip uninstall lightgbm

in the beginning of build-python.sh
and maybe add instruction

sh build-python.sh install

to

https://github.com/microsoft/LightGBM/blob/master/docs/Installation-Guide.rst

as another section:

Build python from source in this repo 

to help future beginners like me

[jameslamb]

Once installed, running build-python.sh script again won't install new light gbm version.

That isn't true (or at least, I'm not able to reproduce that).

Can you please try making that logging change in the relevant file in src/ (not in lightgbm-python/) and running sh build-python.sh install.

[boscocho]

Once installed, running build-python.sh script again won't install new light gbm version.

That isn't true (or at least, I'm not able to reproduce that).

Can you please try making that logging change in the relevant file in src/ (not in lightgbm-python/) and running sh build-python.sh install.

That logging change was made in the relevant file (under src/io/dataset.cpp) in src/ and ran sh build-python.sh install to test if my change was picked up by the build

[jameslamb]

add ... to https://github.com/microsoft/LightGBM/blob/master/docs/Installation-Guide.rst

I don't think we should. The second sentence of that page already says "If you want to build the Python-packagge or R-package please refer to Python-package".

Once installed, running build-python.sh script again won't install new light gbm version

I am not able to reproduce that behavior, and don't think it's an issue with the build script in this project. But I'm glad running pip uninstall fixed it for you!

However, I noticed you're using conda... it is definitely true that sometimes if you mix pip install and conda install, it can result in a state where there are multiple versions of the same library installed. I've seen that before... it's not directly related to LightGBM though.

[jameslamb]

Ah no wait, I see it now! Running build-python.sh multiple times consecutively, I see the following in the logs

Requirement already satisfied: lightgbm in /Users/jlamb/mambaforge/lib/python3.9/site-packages (3.3.5.99)

Hadn't noticed that before. I thought that by "remove original install" you meant "remove an older version installed with pip install".

I'll push a fix for that, thanks for catching it.

[boscocho]

Ah no wait, I see it now! Running build-python.sh multiple times consecutively, I see the following in the logs

Requirement already satisfied: lightgbm in /Users/jlamb/mambaforge/lib/python3.9/site-packages (3.3.5.99)

Hadn't noticed that before. I thought that by "remove original install" you meant "remove an older version installed with pip install".

I'll push a fix for that, thanks for catching it.

Thank you!!!

[boscocho]

add ... to https://github.com/microsoft/LightGBM/blob/master/docs/Installation-Guide.rst

I don't think we should. The second sentence of that page already says "If you want to build the Python-packagge or R-package please refer to Python-package".

Once installed, running build-python.sh script again won't install new light gbm version

I am not able to reproduce that behavior, and don't think it's an issue with the build script in this project. But I'm glad running pip uninstall fixed it for you!

However, I noticed you're using conda... it is definitely true that sometimes if you mix pip install and conda install, it can result in a state where there are multiple versions of the same library installed. I've seen that before... it's not directly related to LightGBM though.

oh I did not realize that the sh ./build-python.sh install is in Python-package already. Will close this question. Thanks so much for quickly help out 👍

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.