Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc] Reorganize documentation on distributed learning (fixes #3596) #3951

Merged
merged 15 commits into from
Feb 22, 2021

Conversation

jameslamb
Copy link
Collaborator

@jameslamb jameslamb commented Feb 13, 2021

This PR finishes #3596 (that work was started in #4000), changing the terminology "parallel learning" or "parallel training" to "distributed learning" and "distributed training".

This also re-organizes the the guide on distributed learning. In this proposed reorganization, the guide would have two top-level sections.

  • "How Distributed LightGBM works": information on the distributed training algorithms. This might be expanded in the future with other information specific to distributed training in general (not any particular programming language or tool).
  • "Integrations": Information on how to run distributed training.

image

In future PRs (for #3814, for example), I'll propose expanding the "integrations --> Dask" section substantially, with code samples and discussion of Dask-specific topics, like how to choose a good partition size and number of partitions, or how to use the Dask package in an environment constrained by firewall rules (#3994 (comment)).

Build Parallel Version
----------------------
Integrations
------------
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -78,11 +111,13 @@ Then write these IP in one file (assume ``mlist.txt``) like following:

**Note**: For Windows users, need to start "smpd" to start MPI service. More details can be found `here`_.

Run Parallel Learning
---------------------
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -1,5 +1,6 @@
Parallel Learning Example
=========================
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -1,24 +1,19 @@
Parallel Learning Guide
=======================
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jameslamb jameslamb changed the title WIP: [doc] Reorganize documentation on distributed learning (fixes #3596) [doc] Reorganize documentation on distributed learning (fixes #3596) Feb 19, 2021
@jameslamb jameslamb marked this pull request as ready for review February 19, 2021 18:18
@jameslamb
Copy link
Collaborator Author

Ok I think this is ready for review!

Docs builds are passing (https://readthedocs.org/projects/lightgbm/builds/). You can explore this PR's docs at https://lightgbm.readthedocs.io/en/docs-jlamb/.

I believe I've accounted for all old links by adding custom anchors. I left comments on each line of the diff that removes a heading, showing a link using the old anchor. If you notice any others that are missed, please let me know.

Copy link
Collaborator

@StrikerRUS StrikerRUS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jameslamb Nice work! Just few comments below.

docs/Parallel-Learning-Guide.rst Outdated Show resolved Hide resolved
docs/Parallel-Learning-Guide.rst Outdated Show resolved Hide resolved
docs/Parallel-Learning-Guide.rst Outdated Show resolved Hide resolved
@jameslamb
Copy link
Collaborator Author

closing and re-opening to retrigger CI, thanks to this infrastructure issue on Azure:

section]This job was abandoned. We have detected that logs from the agent may have not finished uploading. We have included our in-memory record of all log lines uploaded before we lost contact with the agent:
##[section]Starting: Initialize job
Agent name: 'lgbubb5e00003IS'
Agent machine name: 'lgbubb5e00003IS'
Current agent version: '2.182.1'
Agent running as: 'AzDevOps'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: CmdLine (2.182.0)
Downloading task: Bash (3.182.0)

Closing and reopening instead of pushing an empty commit because I'm on mobile right now, and there is currently only one other build in progress.

@jameslamb jameslamb closed this Feb 22, 2021
@jameslamb jameslamb reopened this Feb 22, 2021
Copy link
Collaborator

@guolinke guolinke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@jameslamb jameslamb merged commit 7171558 into master Feb 22, 2021
@jameslamb jameslamb deleted the docs/jlamb branch February 22, 2021 02:43
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants