Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Design of if else op #3828

Merged
merged 9 commits into from
Sep 3, 2017
Merged

Design of if else op #3828

Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ff3ec3c
switch_op.h
zchen0211 Aug 28, 2017
f075141
switch_op.cc
zchen0211 Aug 28, 2017
2f2dd23
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
zchen0211 Sep 2, 2017
d156dfa
if else design doc
zchen0211 Sep 3, 2017
e39a2c9
update if else
zchen0211 Sep 3, 2017
32244b5
if else modified
zchen0211 Sep 3, 2017
d76c60b
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
zchen0211 Sep 3, 2017
523eb23
modify if else
zchen0211 Sep 3, 2017
e0c27a9
if else slight modify
zchen0211 Sep 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions doc/design/if_else_op.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
In an if_op, only inputs with condition satisfied will be run. The op could have multiple inputs and multiple outputs.
We should have the following design:

```python
Copy link
Collaborator

@wangkuiyi wangkuiyi Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basing on #3827, Python bindings of operators and layers should return Vars. So I think that this usage could be

x1 = Var()
x2 = Var()
y1 = Var()
y2 = Var()
cond = Var()

with paddle.block() as left:
  x1 = if_input(x1)
  x2 = if_input(x2)
  y1 = mul(x1, x2)
  y2 = add(x1, x2)

with paddle.block() as right:
  x1 = if_input(x1)
  x2 = if_input(x2)
  y1 = add(x1, x2)
  y2 = mul(x1, x2)

y1, y2 = layer.if(cond, left, right, output=[y1, y2])

Copy link
Contributor

@Superjomn Superjomn Sep 3, 2017 •
edited by wangkuiyi
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

import paddle as pd

x1 = Var()
x2 = Var()
cond = Var()

c1 = pd.IfElseOp(inputs=[x1, x2], output_num=2)
with c1.true_block():
    x1, x2 = c1.inputs()
    y1, y2 = c1.outputs()

    y1 = mul(x1, x2)
    y2 = add(x1, x2)

with c1.false_block():
    x1, x2 = c1.inputs()
    y1, y2 = c1.outputs()

    y1 = add(x1, x2)
    y2 = mul(x1, x2)

out1, out2 = c1(cond)

@wangkuiyi

Copy link
Collaborator

@wangkuiyi wangkuiyi Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @Superjom 's proposal is better than mine because

  1. @Superjom 's approach defines the input and output of the branch block well, and
  2. it doesn't introduce variable duplication/copying -- the x1 and x2 in the block are exactly x1 and x2 defined outside.

Copy link
Collaborator

@wangkuiyi wangkuiyi Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above example breaks the convention that Python operator bindings must return Vars.

However, mine has a severe problem that the two branches would overwrite y1 and y2. A right solution should keep two copies of y1 and y2 for the left and the right branches respectively and then merge these two copies into the global variables y1 and y2.

A correct solution can be derived easily from @Superjom 's solution -- to rename paddle.IfElseOp into paddle.create_ifelseop_builder.

Copy link
Collaborator

@wangkuiyi wangkuiyi Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

import paddle as pd

x1 = var()
x2 = var()

c = pd.create_ifelseop_builder(inputs=[x1, x2], output_num=2)
with c.true_block() as b:
    x1, x2 = b.inputs()
    b.set_output(0, mul(x1, x2))
    b.set_output(1, add(x1, x2))

with c.false_block() as b:
    x1, x2 = c1.inputs()
    b.set_output(0, add(x1, x2))
    b.set_output(1, mul(x1, x2))

cond = var()
out1, out2 = c(cond)

# A 1-d bool vector
cond = Var()
# create an op
if = pd.if_op()

with if.true_block() as block:
x1 = if.input(x1)
x2 = if.input(x2)
y = pd.add(x1, x2)
y2 = pd.fc(x1) # contains (w,b)
if.output(y)
if.output(y2)

o1, o2 = if(cond)
```

In an if_op, only inputs with condition satisfied will be run.
We should have the following design:
```python
Copy link
Collaborator

@wangkuiyi wangkuiyi Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IfOp should have only one branch. An IfOp operator takes a cond variable whose value must be a vector of N boolean elements. Its return value has M (M<=N) instances, each corresponds to a true element in cond.

import paddle as pd

x = var()
y = var()
cond = var()

b = pd.create_ifop_builder(inputs=[x], output_num=1)
with b.true_block():
    x = b.inputs(0)
    z = operator.add(x, y)
    b.set_output(0, operator.softmax(z))

out = b(cond)

If we want the output still has N instances, we can use IfElseOp with a default value, whose minibatch size must be N:

import paddle as pd

x = var()
y = var()
cond = var()
default_value = var()
b = pd.create_ifelseop_builder(inputs=[x], output_num=1, default_value)
with b.true_block():
    x = b.inputs(0)
    z = operator.add(x, y)
    b.set_output(0, operator.softmax(z))

out = b(cond)

If the IfElseOp has multiple return values, the default_value must be a list of variables with corresponding shapes.

# A 1-d bool vector
cond = Var()
# create an op
if = pd.if_op()

with if.true_block() as block:
x1 = if.input(x1)
x2 = if.input(x2)
y = pd.add(x1, x2)
y2 = pd.fc(x1) # contains (w,b)
if.output(y, name="y")
if.output(y2, name="y2")

with if.false_block() as block:
x1 = if.input(x1)
x2 = if.input(x2)
y = pd.fc(x2)
y2 = pd.softmax(x1)
if.output(y, name="y")
if.output(y2, name="y2")

o1, o2 = if(cond)
```

Some questions:
1. how to know which inputs will be selected by condition?
```python
True_block():
y = pd.fc(x)
# we will have x, w, b all as inputs
# but only x will be selected by cond, how can the block know?
```