double_buffer -> concurrent_data_transfer_and_kernel_execution

doc/design/concurrent_data_transfer_and_kernel_execution.md

@@ -0,0 +1,97 @@
+# Design Doc: Concurrent data transfer and kernel execution.
+Training deep learning model on GPU involves three stages: loading data from disk, copying the data from the CPU side to the GPU side, and executing training program on GPU. We need an efficient mechanism to take full use of hardware resources to make those stages executed concurrently. At present, Fluid uses producer-consumer mechanism at the python side to load data from disk. So this design doc mainly solves the time overlap of data transfer and kernel execution.
+
+## Challenge
+
+The data transfer between CPU and GPU is synchronized, by default. A timeline for the execution of training model on `GPU` is shown in the following diagram.
+
+<p align="center"><img src="images/gpu_sync.png" height="75%" width = "75%" align="center"/><br/></p>
+
+In the schematic, we assume that the time required for data transfer and kernel execution is approximately the same. Obviously, the overlap of these operations is zeros. Our target is making use of the hardware resource to parallel data transfer and kernel execution. Just like the following schematic shows.
+
+<p align="center"><img src="images/gpu_async.png" height="75%" width = "75%" align="center"/><br/></p>
+
+## Solution
+One feasible method is to use [staging area](https://en.wikipedia.org/wiki/Staging_(data)). A staging area is an intermediate storage area used for data processing during the [extract, transform and load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load) process. The data staging area sits between the data source(s) and the data target(s), which are often data warehouses, data marts, or other data repositories.   
+The staging area has one fixed-size buffer and two methods, `put` and `get`. `put` and `get` access buffer in a thread-safe way. The operation of `put` puts data into the buffer, and `get` removes a data from the buffer. When the program runs, the staging area should be warmed up. At first, `put` is called to put some data in the staging area. After that, `get` is called when the kernel begins executing, and one data is extracted from the staging area. At the same time, `put` is called and a new data is copied from CPU side to the staging area. Because the operation of data transfer is asynchronous, so the `put` method will return immediately.     
+
+The GPU provided by NVIDIA generally has two engines: a copy engine and a computing engine. To make two engines working at the same time, we need to use the [stream mechanism](https://devblogs.nvidia.com/parallelforall/gpu-pro-tip-cuda-7-streams-simplify-concurrency/) provided by CUDA. As long as data transfer and kernel execution are in different streams and the kernel does not use the data being transferred, the data transfer and kernel execution can be done at the same time on GPU.  
+
+To support the above description, we need to define a new class: `Channel`. Here, we borrow the concept of [`Channel`](https://www.golang-book.com/books/intro/10#section2) in the go language.
+
+```
+template <typename T>
+class Channel {
+ private:
+  using ChannelElement = std::vector<T>;
+
+  std::size_t capacity_;
+  std::size_t bytes_limit_;
+  std::size_t current_bytes_;
+  std::mutex mu_;
+  std::condition_variable empty_cond_var_;
+  std::condition_variable full_cond_var_;
+  std::deque<ChannelElement> channel_;
+
+ public:
+  explicit Channel(std::size_t capacity, std::size_t bytes_limit);
+
+  void Put(ChannelElement* buffer_element) { ... }
+  void Get(ChannelElement* buffer_element) { ... }
+  size_t Size() { ... }
+  void Clear()  { ... }
+};
+};
+```
+
+**Take the [recognize_digits_mlp](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/fluid/tests/book/test_recognize_digits_mlp.py) as an example to show the model definition after using the staging area mechanism.**
+
+```
+...
+concurrent_program = Program()
+with program(concurrent_program):
+    image = data_layer(...)
+    label = data_layer(...)
+    places = get_places()
+    channel_list = create_channel_list(name="data_buffer")
+    with parallel.for(places) as for_loop:
+        cur_channel = create_channel(channel_list, for_loop.i, channel_size=2)  
+        # if channel_list has the specific channel, 
+        # this function will return it directly.
+        send_to_channel(input=[for_loop.input(image), 
+                                for_loop.input(label)], out=cur_channel)
+
+main_program = Program()
+with program(main_program):
+    places = get_places()
+    channel_list = create_channel_list(name="data_buffer")
+    with parallel.for(places) as for_loop:
+        cur_channel = get_channel(channel_list, for_loop.i)
+        image, label = receive_from_channel(input=cur_channel)
+        
+        y_predict = fluid.layers.fc(input=image, size=1, act=None)
+        cost = fluid.layers.square_error_cost(input=y_predict, label=label)
+        avg_cost = fluid.layers.mean(x=cost)
+        ....
+
+for i in range(buffer_size):
+    data = next(train_reader())
+    executor.run(concurrent_program, feed=feeder.feed(data))
+
+for pass_id in range(PASS):
+    for data in train_reader():
+        executor.run(concurrent_program, feed=[...])
+        executor.run(main_program, fetch=[...])
+
+for i in range(buffer_size):
+    executor.run(main_program, fetch=[...])
+
+```
+In Python code, we define two `program`, `concurrent_program` used to send data into `Channel` and `main_program` used to get data from the `Channel` and execute training. If you're familiar with [`Goroutine`](https://www.golang-book.com/books/intro/10#section1) in the go language, you'll find that `main_program` and `concurrent_program` just like two `Goroutine`.
+
+
+
+## Reference
+[Staging area](https://en.wikipedia.org/wiki/Staging_(data))  
+[Producer-Consumer](https://en.wikipedia.org/wiki/Producer–consumer_problem)    
+[How to Overlap Data Transfers in CUDA C/C++](https://devblogs.nvidia.com/parallelforall/how-overlap-data-transfers-cuda-cc/)