[RELAY] BiasAdd, MLP, Resnet testing #1969

tqchen · 2018-10-24T05:49:19Z

This PR recovers the testing API to construct the normal workloads in relay. It contains the following changes

Refactor ir_pass.free_vars to return vars in post-dfs order @MarisaKirisame
- This allows us to directly use this to list unbound vars, which corresponds to old nnvm.symbol.list_arguments
Enable batch_norm to infer the type of the gamma, beta, moving_mean @slyubomirsky
Disable node generic inheritance TupleWrapper
- This is because batch_norm API now returns a tuple, which differs from traditional batch_norm in the inference API
- This is a reasonable change as it makes batch_norm functional and opens for more optimizations
- The decision is to allow the system to eagerly throw when TupleWrapper is mistakenly passed into operators
- For cases when tuple is needed, astuple must be explicitly called.
Introduce relay.testing, which now includes a mlp and resnet

tqchen · 2018-10-24T05:51:42Z

cc @joshpoll @jroesch @yzhliu @zhiics @eqy @merrymercy @Huyuwei please review

MarisaKirisame · 2018-10-24T16:35:26Z

python/tvm/relay/expr.py

@@ -275,6 +296,9 @@ def __repr__(self):
        return ("TupleWrapper(" + self.tuple_value.__repr__() +
                ", " + self.size + ")")

+    def astype(self, _):
+        raise TypeError("astype cannot be used on tuple")


tuple -> tuplewrapper?

MarisaKirisame · 2018-10-24T16:35:37Z

python/tvm/relay/expr.py

+
+        Returns
+        -------
+        result : relay.Expr


tvm.relay.Expr

MarisaKirisame · 2018-10-24T16:35:53Z

python/tvm/relay/op/nn/nn.py

+    bias : relay.Expr
+        The bias to be added.
+
+    axis : int, optional


Optional[int]

MarisaKirisame · 2018-10-24T16:36:12Z

python/tvm/relay/testing/init.py

+    factor_type: str, optional
+        Can be ``'avg'``, ``'in'``, or ``'out'``.
+
+    magnitude: float, optional


Optional[float]

MarisaKirisame · 2018-10-24T16:36:16Z

python/tvm/relay/testing/init.py

+    rnd_type: str, optional
+        Random generator type, can be ``'gaussian'`` or ``'uniform'``.
+
+    factor_type: str, optional


Optional[str]

MarisaKirisame · 2018-10-24T16:36:24Z

python/tvm/relay/testing/init.py

+
+    Parameters
+    ----------
+    net : relay.Function


tvm.relay.Function

MarisaKirisame · 2018-10-24T16:36:36Z

python/tvm/relay/testing/layers.py

+
+    Parameters
+    ----------
+    data : relay.Expr


tvm.relay.Expr

MarisaKirisame · 2018-10-24T16:36:45Z

python/tvm/relay/testing/layers.py

+    data : relay.Expr
+        The input expression.
+
+    weight : relay.Expr


tvm.relay.Expr

MarisaKirisame · 2018-10-24T16:36:54Z

python/tvm/relay/testing/mlp.py

+    image_shape : tuple, optional
+        The input image shape
+
+    dtype : str, optional


Optional[str]

MarisaKirisame · 2018-10-24T16:37:33Z

python/tvm/relay/testing/resnet.py

+        Bottle neck channels factor with regard to num_filter
+    stride : tuple
+        Stride used in convolution
+    dim_match : Boolean


jroesch · 2018-10-24T20:18:54Z

include/tvm/relay/pass.h

@@ -102,35 +102,26 @@ bool AlphaEqual(const Type& t1, const Type& t2);
 */
 bool WellFormed(const Expr& e);

-/*! \brief Get free variables from expression e.
+/*! \brief Get free Vars from expression expr in PostDFS order.


Is expression expr a typo?

jroesch · 2018-10-24T20:20:52Z

Looks good to me 👍, removed similar code from the RTS PR.

tqchen · 2018-10-24T20:51:23Z

Thanks @jroesch @MarisaKirisame for reviews, this is now merged

merrymercy · 2018-10-24T22:17:21Z

python/tvm/relay/testing/init.py

+            raise ValueError("Incorrect factor type")
+        # Hack for mobilenet, because there is less connectivity
+        if "depthwise" in name:
+            factor = 3 * 3


Should we delete this?

merrymercy · 2018-10-24T22:33:28Z

python/tvm/relay/testing/resnet.py

+            strides=stride,
+            padding=(0, 0),
+            name=name + '_conv1')
+        bn2 = layers.batch_norm_infer(data=conv1, epsilon=2e-5, name=name + '_bn2')


These APIs are not as consistent as the old ones in nnvm. Some start with layers., some start with relu., and others start with relu.nn. Some ends with _infer, while the others do not.

We should provide a more consistent API for front-end users.
In this front-end api, we can hide all variable declerations and things like bias_add

True, so far the layers API is a simple extension to the original API and things are for automatic variable creation. As an IR and compiler, we don't need a deep layered API, but I agree that we can use a better API(like gluon) if we want to present a frontend.

I am looking at porting mobilenet; one difference is that it seems before we named relu ops in the graph, but now relay.relu does not take in a name parameter. Not sure if this is a big deal (it seems conv2d and batch norm have wrappers that take in a name parameter).

The name parameter is used to construct weight's name. For now, it is not a big problem in our end

slyubomirsky · 2018-10-24T23:33:45Z

src/relay/op/nn/nn.cc

 bool BatchNormRel(const Array<Type>& types,
                  int num_inputs,
                  const Attrs& attrs,
                  const TypeReporter& reporter) {
  CHECK_EQ(types.size(), 6);
  const auto* data = types[0].as<TensorTypeNode>();
  if (data == nullptr) return false;
-  if (data->shape.size() == 0) return false;


Doesn't this mean that we get an out of bounds if the axis is -1 and the data shape is of rank 0? In that case, axis would be equal to data->shape.size() - 1, or -1, so on line 453, axis_size would be data->shape[-1]. (Sorry for not reviewing before it was merged.)

This is a good catch, I think it is better to directly do a boundary checking (after axis is transformed back by axis + data->shape.size() to avoid this problem

Add initial version of evaluator and tests WIP Work towards simple examples in the evaluator Requires implementation of lowering ops and monomorph Evaluator now works on simple cases Restore Function case in Evaluator WIP Fix rebase issues working towards working version RTS is now working again RTS can add numbers now Fix some rebase issues Fix up tests post rebase WIP Issue type checking MLP Remove dead file Clean up evaluator Remove accidental change Reset changes from apache#1962 Rename Evaluator A little clean up WIP Clean up tests WIP WIP Repair from rebase and refactor to not use MM Remove testing code which is now in apache#1969 WIP

* [RELAY] BiasAdd, MLP, Resnet testing * fix review comments

tqchen force-pushed the master branch from f276ff2 to 224bf55 Compare October 24, 2018 05:51

[RELAY] BiasAdd, MLP, Resnet testing

d45b110

tqchen force-pushed the master branch from a45fe9a to d45b110 Compare October 24, 2018 06:02

MarisaKirisame reviewed Oct 24, 2018

View reviewed changes

jroesch reviewed Oct 24, 2018

View reviewed changes

fix review comments

f713072

tqchen merged commit c76fce9 into apache:master Oct 24, 2018

merrymercy reviewed Oct 24, 2018

View reviewed changes

slyubomirsky reviewed Oct 24, 2018

View reviewed changes

FrozenGene pushed a commit to FrozenGene/tvm that referenced this pull request Dec 27, 2018

[RELAY] BiasAdd, MLP, Resnet testing (apache#1969)

5fa9813

* [RELAY] BiasAdd, MLP, Resnet testing * fix review comments

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY] BiasAdd, MLP, Resnet testing (apache#1969)

a979963

* [RELAY] BiasAdd, MLP, Resnet testing * fix review comments

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY] BiasAdd, MLP, Resnet testing (apache#1969)

66720f7

* [RELAY] BiasAdd, MLP, Resnet testing * fix review comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RELAY] BiasAdd, MLP, Resnet testing #1969

[RELAY] BiasAdd, MLP, Resnet testing #1969

tqchen commented Oct 24, 2018 •

edited

Loading

tqchen commented Oct 24, 2018 •

edited

Loading

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

MarisaKirisame Oct 24, 2018

jroesch Oct 24, 2018

jroesch commented Oct 24, 2018

tqchen commented Oct 24, 2018

merrymercy Oct 24, 2018

merrymercy Oct 24, 2018

tqchen Oct 24, 2018

eqy Oct 24, 2018 •

edited

Loading

tqchen Oct 25, 2018

slyubomirsky Oct 24, 2018

tqchen Oct 25, 2018

[RELAY] BiasAdd, MLP, Resnet testing #1969

[RELAY] BiasAdd, MLP, Resnet testing #1969

Conversation

tqchen commented Oct 24, 2018 • edited Loading

tqchen commented Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jroesch commented Oct 24, 2018

tqchen commented Oct 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eqy Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tqchen commented Oct 24, 2018 •

edited

Loading

tqchen commented Oct 24, 2018 •

edited

Loading

eqy Oct 24, 2018 •

edited

Loading