Adding fluid distributed training guide doc #7619
Conversation
putcn
requested review from
Yancey1989,
helinwang,
gongweibao and
typhoonzero
|
|
||
| ### Have PaddlePaddle installed | ||
|
|
||
| PaddlePaddle must be installed on all nodes. It would be great if you have GPU cards on your nodes, be sure to properly install drivers and CUDA libraries. |
There was a problem hiding this comment.
Maybe change "It would be great if you have GPU cards" to "If you have GPU cards"? Doesn't seem GPU is any better than CPU only nodes in the distributed settings.
There was a problem hiding this comment.
let me update, thanks!
| exit(1) | ||
| ``` | ||
|
|
||
| We created a simple fully connected net program and handled it to the fluid executor to run for 100 passes. |
There was a problem hiding this comment.
Maybe change "net program" to "neural networks training program"
|
|
||
| ## Introduction | ||
|
|
||
| In this article, we'll explain how to config and run distributed training job with PaddlePaddle Fluid in a bare metal cluster. |
|
|
||
| ### Get your cluster ready | ||
|
|
||
| Prepare your computer node in the cluster. Nodes in this cluster can be any spec that runs PaddlePaddle, and with a unique IP address assigned to it. Make sure they can talk to each other. |
There was a problem hiding this comment.
can be any spec -> can be of any specification
There was a problem hiding this comment.
I think it would be better to use the word communicate instead of talk as this is a formal document.
|
|
||
| #### Introducing parameter server | ||
|
|
||
| As you see from the non-cluster version of training script, there is only one role in it: the trainer, who does the computing as well as holding parameters. In cluster training, since multi-trainers are working on the same task, they need one centralized the place to hold and distribute parameters. This centralized place is called Parameter Server in PaddlePaddle. |
There was a problem hiding this comment.
they need one centralized the place -> they need one centralized place
There was a problem hiding this comment.
called Parameter Server -> called the Parameter Server
|
|
||
|  | ||
|
|
||
| Parameter Server in fluid does not only hold parameters but also assigned with part of the program. Trainers communicate with parameter servers via send/receive OPs. For more tech detail, please refer to this [document](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/dist_refactor/distributed_architecture.md). |
There was a problem hiding this comment.
but also -> but is also
with part of the program -> with a part of the program
|
|
||
| Fluid provides a tool called "Distribute Transpiler" to automatically convert the non-cluster program into cluster program. | ||
|
|
||
| The idea behind this tool is to find optimize OPs and gradient parameters, slice the program into 2 pieces and connect then with send/receive OP. |
|
|
||
| The idea behind this tool is to find optimize OPs and gradient parameters, slice the program into 2 pieces and connect then with send/receive OP. | ||
|
|
||
| And optimize OPs and gradient parameters can be found from the return values of optimizer's minimize function. |
| #current_endpoint here means current pserver IP:PORT you wish to run on | ||
| exe.run(t.get_pserver_program(current_endpoint, optimize_ops)) | ||
|
|
||
| # in trianer, run this |
| ... #create executor | ||
|
|
||
| # in pserver, run this | ||
| exe.run(fluid.default_startup_program()) |
There was a problem hiding this comment.
Sorry but the latest transpiler updates the starup_program for each pserver too, you have to run the pserver like below:
pserver_prog = t.get_pserver_program(current_endpoint, optimize_ops)
pserver_startup = t.get_startup_program(current_endpoint, pserver_prog)
exe.run(pserver_startup)
exe.run(pserver_prog)
There was a problem hiding this comment.
got it, let me update the doc.
just to confirm, the trainers will still work with the same default_startup_program right?
fix: #7337