Adding design doc for multi device(background) #5284
Conversation
| On a typical device or system, there are multiple computing devices. In PaddlePaddle, the supported device types as of now, are CPU and GPU (we will be adding FPGA in the future too). | ||
|
|
||
| If a PaddlePaddle operation has both CPU and GPU implementations, we decide which kernel to execute based on the device type. | ||
| Training deep learning models can be resource intensive. Even with a very powerful GPU, some models can take really long to train. This is obvious with deep learning models, especially recurrent models where the execution of each step depends on the execution and output of previous step. |
There was a problem hiding this comment.
Training is a scene of using multi-device. In inference, we may also need to support multi-device. For example, FPGA is used for inference, but FPGA is not suitable for all operators. We may switch to CPU for some complex operators and then, switch back to FPGA.
There was a problem hiding this comment.
Thanks for pointing this out. Will update the writeup.
|
@QingshuChen and @shijiaxin Please kindly review this doc. We are planning to support multi-device feature for Paddle. |
| 2. ProgramDesc: Behind the scenes we need a module that can convert the python API to a ProgramDesc (which is a collection of repeated OpDescs). The ProgramDesc then will be sent to the Executor, which creates the Ops and eventually runs the Ops. | ||
|
|
||
| We need to design the above two components as well as propose how the Python API will be parsed into ProgramDesc. | ||
| These components will be addressed in the following design documents, one for each component. |
There was a problem hiding this comment.
C-API is usually used in inference. We may also need to add C-API document for multi-device support.
There was a problem hiding this comment.
Sounds good. Fixed in the recent commit.
There was a problem hiding this comment.
We will have various combinations of devices. For example
- DGX-1 : x64 CPU + CUDA
- Intel Nervana system: x64 CPU + Xeon Phi
- PX2/3 : ARM CPU + CUDA
- some servers : x64 CPU + FPGA
I agree with @kavyasrinet that we can start from a survey of TensorFlow as the starting point, so could we have a systematic view of challenges and possible solutions.
| ### Model Parallelism | ||
| Model parallelism on the other hand, works by keeping a part of the model on each available device. This is useful when the model is too large to keep on one device or when there are parts of the model that can be executed independently ( in parallel). In this setup, each device will train a part of the model and pass on its updates to the next device. | ||
|
|
||
| Here, we want to explore the model parallelism setup, where different parts of the same model reside on different devices and communicate to each other by sending updates. |
There was a problem hiding this comment.
Here explores model parallelism; does this implies that we prefer model parallelism over data parallelism?
There was a problem hiding this comment.
As per my discussion with @QiJune , he mentioned that we should focus on model parallelism at the moment, some other team is already looking at data parallelism. If that's not the case, I would be happy to include more about data parallelism as well.
There was a problem hiding this comment.
Hello, I think maybe computation node(operator) placement policy is another topic that is worth mentioning in multi-device situation.
Sometimes the model is not parallel, but it still benefit the overall training and inference efficiency to put some compute nodes on devices like GPU or FPGA, like DNN/CNN part. Of course user could be able to explicitly specify on which device each of the model's node is computed, but I think how to assign each node's computing device automatically is worth discussing.
There was a problem hiding this comment.
Thanks for pointing that out, I covered how TensorFlow does that in this PR: #5412 , but this is a good suggestion, I will fix this PR to include this too.
There was a problem hiding this comment.
Hello @zealoct , I have fixed the changes you proposed in the latest commit. Please have a look when you can.
This is the design document that explains the background of multi-device training.
The next steps are: