[ci] isolate c_api_test library-loading from Python source tree

[jameslamb]

Contributes to #5061

This repo contains a few tests on LightGBM's C API which are run with pytest.

Today, they load lib_lightgbm.{dll/so} using a modified copy of the same library-loading code with ctypes used in the Python package. LightGBM's CI runs python setup.py bdist_wheel, then these C API tests rely on a lib_lightgbm.{dll/so} being available in python-package/compile/.

LightGBM/tests/c_api_test/test_.py

Lines 17 to 29 in 709ea4c

	dll_path = [curr_path,
	curr_path.parents[1],
	curr_path.parents[1] / 'python-package' / 'lightgbm' / 'compile',
	curr_path.parents[1] / 'python-package' / 'compile',
	curr_path.parents[1] / 'lib']
	if system() in ('Windows', 'Microsoft'):
	dll_path.append(curr_path.parents[1] / 'python-package' / 'compile' / 'Release/')
	dll_path.append(curr_path.parents[1] / 'python-package' / 'compile' / 'windows' / 'x64' / 'DLL')
	dll_path.append(curr_path.parents[1] / 'Release')
	dll_path.append(curr_path.parents[1] / 'windows' / 'x64' / 'DLL')
	dll_path = [p / 'lib_lightgbm.dll' for p in dll_path]
	else:
	dll_path = [p / 'lib_lightgbm.so' for p in dll_path]

This PR proposes removing that reliance on the repo structure + exactly where lib_lightgbm.so copies get left behind after building Python wheels.

How does this contribute to #5061? The changes for that (in-progress in #5759) will involve building lightgbm wheels in an isolated build directory, which breaks the assumptions in c_api_test/test_.py that a lib_lightgbm will be left behind in the python-package/ directory.

How I tested this

Check that this works with the Python package installed.

pip uninstall --yes lightgbm
cd python-package/
python setup.py bdist_wheel
pip install dist/*.whl
cd ..
pytest tests/c_api_test

Check that it works without the Python package.

pip uninstall --yes lightgbm
rm -rf ./python-package/build
rm -rf ./python-package/compile
rm -rf ./python-package/dist
mkdir build
cd build
cmake ..
make -j3
cd ..
pytest tests/c_api_test

[jameslamb]

@guolinke or @shiyu1994 can I please get a review on this this week? It's important for enabling the work to change the strategy for building Python wheels (#5061).

Thank you!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.