From fff380b54e1d8da774e0c183ad206ffbac5f9b4b Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:03:45 -0700 Subject: [PATCH 1/8] Add layers and operators design doc --- doc/design/layers_operators.md | 71 ++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 doc/design/layers_operators.md diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md new file mode 100644 index 0000000000000..4cb590c85f976 --- /dev/null +++ b/doc/design/layers_operators.md @@ -0,0 +1,71 @@ +In a DL system, we can compose one or more fine grained operators into a coarse grained one. For example, the FC layer can be composed of a multiplication operator and an add operator. + +Historically, some fine grained operations are known as operators, and some coarse level ones are known as layers. But we need a well-defined separation. + +In general, operators are those very fine grained operations, e.g., mul and add. In the implementation, we can write them as C++ functions: + +```c++ +template T add(T x, T y) { return x + y; } +template T mul(T x, T y) { return x * y; } +``` + +Then we can wrap them into operators which are C++ classes and can be created from Python bindings by name. A C macro can do this. For example, the following macro invocation + +```c++ +#define MAKE_FUNCTION_OPERATOR(mul); +``` + +generates + +```c++ +class mulOp : public OperatorBase {...}; +REGISTER_OP(mulOp, "mul"); +``` + +so that in Python we can create operator mul by: + +```python +X1 = Var() +X2 = Var() +Y = Var() +paddle.cpp.create_operator("mul", input=[X1, X2], output=Y) +``` + +Also, at the same time, we can compose a coarse level C++ operator class by composing functions `mul` and `add`: + +```c++ +class FCOp : public OperatorBase { + public: + void Run(...) { + add(mul(Input("X"), Input("W")), Input("b"); + } +}; +REGISTER_OP(FCOp, "fc"); +``` + +We need to support such composition in Python as well. To do so, we need a higher level Python wrapping of operator creation than `paddle.cpp.create_operator`. This higher level operator API should be compatible with the layer API. + +Let's explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we'd like to have Python functions `mul` and `add` defined in module `operator`: + +```python +def mul(X1, X2): + O = Var + paddle.cpp.create_operator("mul", input={X1, Y1], output=O) + return O + +def add(X1, X2): + O = Var + paddle.cpp.create_operator("add", input={X1, X2], output=O) + return O +``` + +so that we can define + +```python +def layer.fc(X): + W = Var() + b = Var() + return operator.add(operator.mul(X, W), b) +``` + +We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. This is how we differentiate layer and operators in PaddlePaddle. From 6aef11baac1ff09b0fc69dd801ba1fe508374c31 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:05:37 -0700 Subject: [PATCH 2/8] Add title --- doc/design/layers_operators.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md index 4cb590c85f976..83169260c8f5f 100644 --- a/doc/design/layers_operators.md +++ b/doc/design/layers_operators.md @@ -1,3 +1,5 @@ +# Design Doc: Layers and Operators + In a DL system, we can compose one or more fine grained operators into a coarse grained one. For example, the FC layer can be composed of a multiplication operator and an add operator. Historically, some fine grained operations are known as operators, and some coarse level ones are known as layers. But we need a well-defined separation. From 887275efad3e9e7184b17ee6a141196756d8fc07 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:12:24 -0700 Subject: [PATCH 3/8] Add a table --- doc/design/layers_operators.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md index 83169260c8f5f..ee9e890e5cec4 100644 --- a/doc/design/layers_operators.md +++ b/doc/design/layers_operators.md @@ -50,12 +50,12 @@ We need to support such composition in Python as well. To do so, we need a high Let's explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we'd like to have Python functions `mul` and `add` defined in module `operator`: ```python -def mul(X1, X2): +def operator.mul(X1, X2): O = Var paddle.cpp.create_operator("mul", input={X1, Y1], output=O) return O -def add(X1, X2): +def operator.add(X1, X2): O = Var paddle.cpp.create_operator("add", input={X1, X2], output=O) return O @@ -70,4 +70,14 @@ def layer.fc(X): return operator.add(operator.mul(X, W), b) ``` -We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. This is how we differentiate layer and operators in PaddlePaddle. +We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. So we have the following concepts in above illustrative example: + +| C++ functions/functors | mul | add | | | +| C++ operator class | mulOp | addOp | FCOp | | +| Python binding | operator.mul | operator.add | operator.fc | | +| Python function | | | | layer.fc | + +This is how we differentiate layer and operators in PaddlePaddle: + +- those defined in C++ and have a lightweighted Python wrapper in module `operators` are operators; whereas +- those who don't have C++ implementations but a Python implementation that compose C++ operators are known as layers. From 7159062873bc097e9a9951eb009e5f625b23b4a9 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:13:37 -0700 Subject: [PATCH 4/8] Add table header --- doc/design/layers_operators.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md index ee9e890e5cec4..cda99888bea6d 100644 --- a/doc/design/layers_operators.md +++ b/doc/design/layers_operators.md @@ -72,6 +72,8 @@ def layer.fc(X): We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. So we have the following concepts in above illustrative example: +| implementation | mul | add | fc | fc | +--------------------------------------------------------------------------------- | C++ functions/functors | mul | add | | | | C++ operator class | mulOp | addOp | FCOp | | | Python binding | operator.mul | operator.add | operator.fc | | From 90a442825c42740173ecf5ee3e64c24756bcfd56 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:14:51 -0700 Subject: [PATCH 5/8] Change table to verbatim --- doc/design/layers_operators.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md index cda99888bea6d..131effbccff5d 100644 --- a/doc/design/layers_operators.md +++ b/doc/design/layers_operators.md @@ -72,12 +72,12 @@ def layer.fc(X): We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. So we have the following concepts in above illustrative example: -| implementation | mul | add | fc | fc | ---------------------------------------------------------------------------------- +``` | C++ functions/functors | mul | add | | | | C++ operator class | mulOp | addOp | FCOp | | | Python binding | operator.mul | operator.add | operator.fc | | | Python function | | | | layer.fc | +``` This is how we differentiate layer and operators in PaddlePaddle: From 1f713f6822b96ccaff5684e547dac32590047d85 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:25:05 -0700 Subject: [PATCH 6/8] in response to comments from Jia-Yi --- doc/design/layers_operators.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md index 131effbccff5d..e4f8d303e8b55 100644 --- a/doc/design/layers_operators.md +++ b/doc/design/layers_operators.md @@ -1,4 +1,4 @@ -# Design Doc: Layers and Operators +# Design Doc: Functions, Operators, and Layers In a DL system, we can compose one or more fine grained operators into a coarse grained one. For example, the FC layer can be composed of a multiplication operator and an add operator. @@ -20,8 +20,8 @@ Then we can wrap them into operators which are C++ classes and can be created fr generates ```c++ -class mulOp : public OperatorBase {...}; -REGISTER_OP(mulOp, "mul"); +template class mulOp : public OperatorBase {...}; +REGISTER_OP(mulOp, "mul"); ``` so that in Python we can create operator mul by: @@ -36,12 +36,13 @@ paddle.cpp.create_operator("mul", input=[X1, X2], output=Y) Also, at the same time, we can compose a coarse level C++ operator class by composing functions `mul` and `add`: ```c++ +template class FCOp : public OperatorBase { public: void Run(...) { - add(mul(Input("X"), Input("W")), Input("b"); + add(mul(Input("X"), Input("W")), Input("b"); } -}; +};d REGISTER_OP(FCOp, "fc"); ``` From 305aebc005c4083ee3733688fc1ed23b5e1a8232 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:26:44 -0700 Subject: [PATCH 7/8] Rename file according to title --- doc/design/{layers_operators.md => functions_operators_layers.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/design/{layers_operators.md => functions_operators_layers.md} (100%) diff --git a/doc/design/layers_operators.md b/doc/design/functions_operators_layers.md similarity index 100% rename from doc/design/layers_operators.md rename to doc/design/functions_operators_layers.md From 2b3904a6a8c568f8bf20155f8f87a33c9f4e67b9 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sat, 2 Sep 2017 17:34:10 -0700 Subject: [PATCH 8/8] Explain auto code generation --- doc/design/functions_operators_layers.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/doc/design/functions_operators_layers.md b/doc/design/functions_operators_layers.md index e4f8d303e8b55..7a2e8fd0ace2e 100644 --- a/doc/design/functions_operators_layers.md +++ b/doc/design/functions_operators_layers.md @@ -42,7 +42,7 @@ class FCOp : public OperatorBase { void Run(...) { add(mul(Input("X"), Input("W")), Input("b"); } -};d +}; REGISTER_OP(FCOp, "fc"); ``` @@ -52,17 +52,17 @@ Let's explain using an example. Suppose that we are going to compose the FC usi ```python def operator.mul(X1, X2): - O = Var + O = Var() paddle.cpp.create_operator("mul", input={X1, Y1], output=O) return O def operator.add(X1, X2): - O = Var + O = Var() paddle.cpp.create_operator("add", input={X1, X2], output=O) return O ``` -so that we can define +Above code snippets are automatically generated. Given them, users can define ```python def layer.fc(X): @@ -71,6 +71,19 @@ def layer.fc(X): return operator.add(operator.mul(X, W), b) ``` +If we don't have `operator.mul` and `operator.add`, the definiton of `layer.fc` would be complicated: + +```python +def layer.fc(X): + W = Var() + b = Var() + O1 = Var() + paddle.cpp.create_operator("mul", input=[X, W], output=O1) + O2 = Var() + paddle.cpp.create_operator("add", input=[O1, b], output=O2) + return O2 +``` + We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`. So we have the following concepts in above illustrative example: ```