Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] run Dask examples on CI #4064

Merged
merged 5 commits into from
Mar 15, 2021
Merged

[ci] run Dask examples on CI #4064

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
eccd807
Update Parallel-Learning-Guide.rst
StrikerRUS Mar 12, 2021
841990c
Update test.sh
StrikerRUS Mar 12, 2021
cf324d2
fix path
StrikerRUS Mar 12, 2021
ac96c25
Merge branch 'master' into dask_examples
StrikerRUS Mar 14, 2021
b2c1685
address review comments
StrikerRUS Mar 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ci/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ import matplotlib\
matplotlib.use\(\"Agg\"\)\
' plot_example.py # prevent interactive window mode
sed -i'.bak' 's/graph.render(view=True)/graph.render(view=False)/' plot_example.py
for f in *.py; do python $f || exit -1; done # run all examples
for f in *.py **/*.py; do python $f || exit -1; done # run all examples
cd $BUILD_DIRECTORY/examples/python-guide/notebooks
conda install -q -y -n $CONDA_ENV ipywidgets notebook
jupyter nbconvert --ExecutePreprocessor.timeout=180 --to notebook --execute --inplace *.ipynb || exit -1 # run all notebooks
Expand Down
6 changes: 5 additions & 1 deletion docs/Parallel-Learning-Guide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,14 @@ Dask

LightGBM's Python package supports distributed learning via `Dask`_. This integration is maintained by LightGBM's maintainers.

.. warning::

Dask integration is only tested on Linux.

Dask Examples
'''''''''''''

For sample code using ``lightgbm.dask``, see `these Dask examples`_ .
For sample code using ``lightgbm.dask``, see `these Dask examples`_.

Training with Dask
''''''''''''''''''
Expand Down
1 change: 0 additions & 1 deletion examples/python-guide/dask/binary-classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import lightgbm as lgb

if __name__ == "__main__":

print("loading data")

X, y = make_blobs(n_samples=1000, n_features=50, centers=2)
Expand Down
1 change: 0 additions & 1 deletion examples/python-guide/dask/multiclass-classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import lightgbm as lgb

if __name__ == "__main__":

print("loading data")

X, y = make_blobs(n_samples=1000, n_features=50, centers=3)
Expand Down
1 change: 0 additions & 1 deletion examples/python-guide/dask/prediction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import lightgbm as lgb

if __name__ == "__main__":

print("loading data")

X, y = make_regression(n_samples=1000, n_features=50)
Expand Down
9 changes: 6 additions & 3 deletions examples/python-guide/dask/ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import os

import dask.array as da
import numpy as np
from distributed import Client, LocalCluster
Expand All @@ -6,11 +8,12 @@
import lightgbm as lgb

if __name__ == "__main__":

print("loading data")

X, y = load_svmlight_file("../lambdarank/rank.train")
group = np.loadtxt("../lambdarank/rank.train.query")
X, y = load_svmlight_file(os.path.join(os.path.dirname(os.path.realpath(__file__)),
'../../lambdarank/rank.train'))
group = np.loadtxt(os.path.join(os.path.dirname(os.path.realpath(__file__)),
'../../lambdarank/rank.train.query'))

print("initializing a Dask cluster")

Expand Down
1 change: 0 additions & 1 deletion examples/python-guide/dask/regression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import lightgbm as lgb

if __name__ == "__main__":

print("loading data")

X, y = make_regression(n_samples=1000, n_features=50)
Expand Down
4 changes: 4 additions & 0 deletions python-package/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,10 @@ You can use ``python setup.py bdist_wheel`` instead of ``python setup.py install
Install Dask-package
''''''''''''''''''''

.. warning::

Dask-package is only tested on Linux.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this file is viewed on GitHub, this warning doesn't render the way it would in Sphinx.

image

Are you ok with that? It doesn't bother me because I hope we'll be able to remove this soon anyway, but just wanted you to be aware.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, like all other Warnings and Notes in other RSTs:

image

I believe there should be no problem with rendering on PyPI because http://rst.ninjs.org/ shows no errors when I paste raw content of python-package/README.rst.
FYI, PyPI used to show blank page in case of any errors in RST formatting of uploaded README, but I don't know what's the current policy of error handling there.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok sounds good!


To install all additional dependencies required for Dask-package, you can append ``[dask]`` to LightGBM package name:

.. code:: sh
Expand Down